Add references to Q&A apps
Include references to your knowledgebase in a Questioning and Answering app
With Relevance, it’s very easy to create a Q&A app on your knowledgebase using our Knowledge feature. However, a common question we’re asked is how to include references in the response. In this example, we’ll take a look at a couple of approaches you can take.
Adding fields to use as a reference
To be able to create references, you need to make sure the context you provide to the LLM includes a field that you can use as a reference. For example, this typically is a URL, a filename, an article title or a product name.
If you’re uploading a CSV to Knowledge, this will mean you’ll need to ensure you have a column that will act as your identifier that will be used in the reference. If you’re upload a PDF or an audio file, then a column with the filename will be automatically created.
In this example, we’ll use a CSV that has columns “title”, “content” and “url” where each row is a different article from our support documentation.
Configuring your Knowledge to include the correct fields
Under “Advanced options” in the LLM step, you will find a section on “How to handle too much text”. If you click “edit” for the knowledge
variable you will see a modal that has an “Advanced options” section. Under here, you will find an input to configure which columns from the dataset to include. Specify the columns that include the knowledge and the ones that you’ll use as reference here.
In this example, we’ll include “title”, “content” and “url”. Now, when we use the knowledge
variable in our LLM, it’ll be replaced with only these fields from the most relevant content.
Screenshot of the column selection modal
Updating your prompt to include the reference
Now that we have configured our knowledge to include the fields we want, we can update our prompt to include the reference. In this example, we’ll use the url
field as our reference. There are many different ways you can choose to format this, here we cover two: JSON and Footnotes.
Specify the response as JSON in format { answer, references: [{title, url}]" }. Use proper punctuation. Never provide URLs that are not included in my messages. You must respond in JSON.
We have asked for a JSON response that includes an answer
and a references
field. The references
field is an array of objects that have a title
and a url
field. The title
field is the title of the article and the url
field is the URL of the article.
Adjust the prompts to suit your data, output and accuracy needs best. LLMs are more than capable of providing references and can even be extend to include things like quotes.
Was this page helpful?