Introduction

What does it take to turn an application that can answer questions like:

“How many employees does Microsoft have?”

…into one that can automate tasks like:

“Create a plot of the number of employees at the three tech companies with the highest market cap in the United States in 2024.”?

The answer is to transition from a retrieval-augmented generation (RAG) to a tool use approach.

Suppose we are asking a human assistant to perform these two tasks. For the first task, simply entering the exact question into a web search engine would provide the answer.

As for the second task, they would have to:

Break down the task into smaller parts and perform them in a sequence (first, find the three companies, and only then find the employee number for each)
Perform an action multiple times (multiple web searches)
Not only perform information retrieval (web search) but also perform some kind of action (creating a plot)

This planning, and then the execution of every task, can be performed with tool use.

Tool use can automate the tasks required to fulfill a user's request

In this module, we’ll learn how to leverage tool use by building:

A RAG assistant that can query structured data about sales and product
A calendar assistant that can check and schedule appointments
A data analyst that’s capable of answering questions like the above

RAG

To understand what tool use is, it’s helpful to start by understanding RAG. We’ll only do a brief recap here as we covered RAG in detail in Module 7.

RAG gives LLMs the capability to ground their responses in external text data. This means we can build applications that are more robust against hallucination, can access recent events, and are personalized to an enterprise’s data and context.

With RAG, an LLM retrieves unstructured text documents from external data sources to provide useful and accurate responses to user queries.

RAG utilizes text documents from external data sources

A common setup is to store these documents in a vector database as text embeddings. During query time, we use search techniques, such as semantic search, to retrieve the documents most relevant to a user's query.

These documents are then augmented to the LLM prompt, acting as the context to inform and guide the LLM's response to the user.

The document retrieval system can be set up for any source that contains textual, unstructured data. Examples would be a company's meeting notes stored on Google Drive, its internal wiki stored in Notion, its customer conversations on Intercom, PDF documents, and so on.

In a RAG workflow, documents are used as the context to guide an LLM's response — In a RAG workflow, documents are used to guide an LLM's response

Tool Use

Tool use takes the RAG concept to the next level. It extends the principles of RAG, where external systems are used to guide the response of an LLM, but by leveraging a much bigger set of tools than what’s possible with RAG — as we’ll see in this module.

With the RAG example earlier, we are already using a type of tool — the document retriever. But tools can come in many forms beyond just that.

If there is a way to communicate and send information to and from a service via an API, that service can become a tool for an LLM to utilize. It can be a web search engine, a SQL database, a weather forecast service, or a Python interpreter, to name a few.

So, instead of retrieving information from unstructured documents (typical RAG), we can retrieve information from any source (tool use). Plus, we'll see later on that this tool use setup allows us to not only retrieve information from external tools, but also take actions through them.

In a tool use workflow, results from tool calls are used to guide an LLM's response

Decision-Making

The concept of tool use expands on LLMs' capabilities of acting as a reasoning and decision-making engine.

With a standard RAG setup, an LLM needs to make one type of decision. Given a user query, it has to decide whether it needs external data to respond or can respond directly. With Cohere’s Chat endpoint, this is done using the search query generation feature.

For example, "How do I build a great company?" doesn't usually require retrieving a specific piece of data, but "What was Apple's revenue in 2023?" does.

With tool use, the model needs to make two types of decisions.

The first is similar to RAG, like in the example above, which is a yes/no decision on whether a user query can be answered directly or requires some external context
The second is if tools are required, which tools would those be (and how to call them)

In a tool use setting, a model is often given several tools to pick from. For example, let's say that those are a web search engine and a Python interpreter.

So, if a user asks, "What were Apple's and Google's revenue in 2023?" Then, the model would determine that only the web search engine is needed.

But if the question is "What were Apple's and Google's revenue in 2023, and plot them on a bar chart?" then the model would determine that both the web search engine and the Python interpreter are needed.

This concept of routing a request to only the relevant tools can enhance the quality of a model’s response. In a naive RAG setting, the application would aggregate retrieved documents from all the different sources it is connected to. This may contribute to noise from less relevant documents.

Additionally, it doesn’t take into consideration that, given a data source's nature, it might be less or more relevant to a query compared to the other data sources.

For example, suppose an application is connected to two data sources, one to access public data (the web) and the other to private data (a company’s internal Notion knowledge base). Given the question of “Who were the largest car manufacturers in 2023?”, the information is more likely to be available publicly, and it makes more sense to search just the web rather than both.

A naive RAG system does not offer this routing capability. Moving to a tool use paradigm solves this.

Tool use enables routing a request to only the relevant tools

Action

A standard RAG setup is read-only. The RAG application can only retrieve documents from the source (or a read action), but it doesn't need to make any changes in the opposite direction (or a write action).

Tool use enables developers to build applications that can take actions. That is, doing both read and write operations on an external system.

For example, if it is a calendar service, the tool use approach not only enables the application to look up a person’s availability (read) but also create new events on the calendar (write).

The ability to read, write, and call APIs means that an LLM is more connected to the external world, can take a broader range of actions, and is more flexible overall.

Tool use enables LLMs to be more connected to the external world

Function Calling

If tool use is the what of LLM capability extension, function calling is the how.

While tool use is a broader concept that encompasses the use of various external systems and tools to enhance LLM capabilities, function calling, on the other hand, is a specific implementation of tool use, focusing on the technical details of picking the tools to call and generating the payload to call the tool.

Function calling is the process of tool selection and payload generation to perform specific tasks within a program. It is a standardized way for an LLM to interface with tools to send and receive information.

Every tool can be represented as a function with a schema that informs the LLM about what the tool does, what parameters the tool accepts, what these parameters do, and the types of the parameters. These are used by the LLM to make tool use decisions. We’ll go deeper into function calling in the coming chapters.

Function calling is a specific implementation of tool use

Citations

Tool use is such a natural extension of RAG that many concepts we see in RAG can be ported over to the tool use scenario.

One such way is generating citations. A big part of building a successful RAG application is implementing a citation mechanism to trace LLM responses back to the source documents. This allows an LLM’s responses to be easily verifiable, building trust with the user.

In the same way, you need the same citation mechanism when tool results are used to augment an LLM’s response. The range of tools is now much wider, but the need to trace the response to its source remains.

Through Cohere’s Command R family of models, you get citation generation as a built-in feature in RAG without having to do prompt engineering or fine-tuning. And the same goes for tool use!

When using Command R/R+ to generate responses, it will generate fine-grained citations referencing the tool results being used to ground its response. Here, fine-grained means specific spans of the text generated based on results from one or more tools — more specific than just at the response or even sentence level.

Cohere's Command R/R+ models produce fine-grained citations to their responses

Conclusion

This was a quick introduction to the concept of tool use, starting with using RAG as the starting point. In the coming chapters, we’ll look at hands-on examples and concrete use cases of how tool use is applied in actual applications.

In Chapter 2, we’ll look at the building blocks of a tool use system and how they all work together in a workflow. We’ll also look at our first code example by building a RAG assistant that can query structured sales data.

Contents

From RAG to Tool Use