While LLMs are good at maintaining the context of the conversation and generating responses, they can be prone to hallucinate and include factually incorrect or incomplete information in their responses.

Retrieval-augmented generation (RAG) is a technique that enhances the performance of LLMs by incorporating external data sources. This approach significantly reduces the hallucination issue common in LLMs. RAG enables the model to access and utilize supplementary information from external documents, thereby improving the accuracy of its responses.

The chatbot provides helpful and verifiable responses through citations

Having RAG in a chat paradigm means you can build context-aware applications that are able to both maintain the state of a conversation and generate grounded responses.

The Chat endpoint adds RAG capabilities to the chat paradigm

RAG with Cohere

The Cohere Chat endpoint comes with RAG features already integrated. This greatly simplifies the task of developing RAG-powered applications.

With Cohere Chat, you get the complete set of building blocks needed to build a high-quality RAG application in the shortest time possible. We’ll cover them in depth throughout this module, but first, let’s take a quick look at some key capabilities of Cohere’s RAG solution.

Chat interface: The RAG functionalities run on the Chat endpoint. That means everything is wrapped in a chat interface and powered by the Command model. Thus, you can build chatbots that have the full context of a conversation and are not limited to a single interaction.
Response generation: Cohere’s RAG solution gives you an LLM that can provide the right responses to the user in different scenarios. A good RAG system should generate a grounded response based on relevant documents, but it should not do that every single time. The system also has to be able to determine whether or not any of the provided documents are relevant (and possibly decide that none are relevant), as well as decide that it can directly respond without needing any documents retrieved.
Fine-grained citation: Each grounded response includes fine-grained citations linking back to the source documents. This makes the response easily verifiable and builds trust with the user.

Step-by-Step Guide

Let’s start our exploration of RAG with a quick example.

We’ll walk through how to ground an LLM’s response with information from external documents and provide document citations along with it. In this example, we’ll use a static, short list of documents. Below is a diagram that provides an overview of our simple RAG system.

Setup

First, let’s install and import the cohere library, and then create a Cohere client using an API key.

pip install cohere

import cohere
co = cohere.ClientV2("COHERE_API_KEY")

Define the Documents

Next, we define the documents that we want to ground an LLM’s response with, formatted as a list. In the V2 API, each document includes an id and a data object. In our case, the data object contains two fields: title and text.

The documents list includes a list of documents with a text field containing the information we want the model to use. The recommended length for the snippet of each document is relatively short, 300 words or less. We recommend using field names similar to the ones we’ve included in this example (i.e., title and text), but RAG is quite flexible with respect to how you structure the document data. You can give the fields any names you want, and you can pass in other fields as well, such as a date field. All field names and field values in data are passed to the model.

documents = [
    {
        "id": "doc_0",
        "data": {
            "title": "Tall penguins",
            "text": "Emperor penguins are the tallest.",
        },
    },
    {
        "id": "doc_1",
        "data": {
            "title": "Penguin habitats",
            "text": "Emperor penguins only live in Antarctica.",
        },
    },
    {
        "id": "doc_2",
        "data": {
            "title": "What are animals?",
            "text": "Animals are different from plants.",
        },
    },
]

Generate the Response with Citations

Cohere’s RAG functionalities are part of the Chat endpoint, with the Command model as the underlying LLM. This allows developers to build chatbots that have the full context of a conversation and are not limited to a single interaction.

First, we define the message coming from the user. We’ll use a simple query, “What are the tallest living penguins?”, as an example.

# Get the user message
message = "What are the tallest living penguins?"

Then, we pass this message to the Chat endpoint using the V2 messages format. We also pass the list of documents as a documents parameter.

# Generate the response
response = co.chat(
    model="command-a-plus-05-2026",
    messages=[{"role": "user", "content": message}],
    documents=documents,
)

Finally, we display the response from the model. We extract only text content items because newer models can also return non-text content items.

We also display the citations and source documents from response.message.citations.

# Display the response
text = "".join(item.text for item in response.message.content if item.type == "text")
print(text)

# Display the citations and source documents
if response.message.citations:
    print("\n\nCITATIONS:")
    for citation in response.message.citations:
        print(citation)

    print("\nDOCUMENTS:")
    seen = set()
    for citation in response.message.citations:
        for source in citation.sources:
            if source.type == "document" and source.id not in seen:
                seen.add(source.id)
                print(source.document)

And here’s the response generated by our RAG system.

According to the document, Emperor penguins are the tallest living penguins.

CITATIONS:
start=27 end=76 text='Emperor penguins are the tallest living penguins.' sources=[DocumentSource(type='document', id='doc_0', document={'id': 'doc_0', 'text': 'Emperor penguins are the tallest.', 'title': 'Tall penguins'})] content_index=1 type='TEXT_CONTENT'

DOCUMENTS:
{'id': 'doc_0', 'text': 'Emperor penguins are the tallest.', 'title': 'Tall penguins'}

First, we get the actual text response from the model (see the output below).

This is followed by a list of citations, which are references to specific source documents on our list that provided the information contained in specific passages within the text response. For example, the citation indicates that the generated answer was grounded in the first document on the list (doc_0).

Finally, we get the full list of the source documents used to generate the response.

Getting Started with Retrieval-Augmented Generation