We’ll use Cohere’s Python SDK for the code examples. Follow along in this notebook.

Introduction

In the previous chapter, we saw that tool use can run in two different modes: single-step and multi-step.

In a single-step tool use scenario, an LLM calls a single tool or many tools in parallel. The tools then return the results, which the LLM uses to generate its response.

In single-step tool use, an LLM calls a single tool or many tools in parallel

We also went through a code example of single-step tool use. However, that example was a basic one. We haven’t yet seen these cases:

Multiple tools: The example only had one tool, but what if the model had multiple tools at its disposal? How does the model reason over which tools to use at a particular time?
Parallel tool calls: We had only seen a single tool call in a single step, but what does it look like if more than one tool call is required?
When not to use tools: What if a question can, and should, be answered directly by a model without needing a tool?
Tool use in a chat setting: What if the user had follow-up questions, i.e., how does the assistant maintain the context of a conversation in a multi-turn chat setting?

In this chapter, let’s go deeper into single-step tool use and address the questions above.

Setting Up the Tools

Let’s first set up the Cohere client. We’ll also need to import the json module for use later.

# pip install cohere

import json
import cohere
co = cohere.Client(api_key="YOUR_COHERE_API_KEY") # Get your API key: https://dashboard.cohere.com/api-keys

Step 1: Create Tools

Let’s now create two tools:

A function to query the sales database called query_daily_sales_report, which includes a mock sales database (same as the previous chapter)
A function to query the product catalog called query_product_catalog, which includes a mock product catalog (a new one)

def daily_sales_report(day: str) -> dict:
    """
    Function to retrieve the sales report for the given day
    """
    # Mock database containing daily sales reports
    sales_database = {
    '2023-09-28': {'total_sales_amount': 5000,'total_units_sold': 100},
    '2023-09-29': {'total_sales_amount': 10000,'total_units_sold': 250},
    '2023-09-30': {'total_sales_amount': 8000,'total_units_sold': 200}
    }
    
    report = sales_database.get(day, {})
    
    if report:
        return {
            'date': day,
            'summary': f"Total Sales Amount: {report['total_sales_amount']}, Total Units Sold: {report['total_units_sold']}"
        }
    else:
        return {'date': day, 'summary': 'No sales data available for this day.'}
    

def product_database(category: str) -> dict:
    """
    Function to retrieve products for the given category
    """
    
    # Mock product catalog
    product_catalog = {
        'Electronics': [
            {'product_id': 'E1001', 'name': 'Smartphone', 'price': 500, 'stock_level': 20},
            {'product_id': 'E1002', 'name': 'Laptop', 'price': 1000, 'stock_level': 15},
            {'product_id': 'E1003', 'name': 'Tablet', 'price': 300, 'stock_level': 25},
        ],
        'Clothing': [
            {'product_id': 'C1001', 'name': 'T-Shirt', 'price': 20, 'stock_level': 100},
            {'product_id': 'C1002', 'name': 'Jeans', 'price': 50, 'stock_level': 80},
            {'product_id': 'C1003', 'name': 'Jacket', 'price': 100, 'stock_level': 40},
        ]
    }
    
    products = product_catalog.get(category, [])
    return {
        'category': category,
        'products': products
    }

functions_map = {
    "daily_sales_report": daily_sales_report,
    "product_database": product_database
}

Step 2: Define Tool Schema

Next, we define the tool schema for the two tools. Both functions accept one parameter, day and category respectively.

tools = [
    {
        "name": "daily_sales_report",
        "description": "Connects to a database to retrieve overall sales volumes and sales information for a given day.",
        "parameter_definitions": {
            "day": {
                "description": "Retrieves sales data for this day, formatted as YYYY-MM-DD.",
                "type": "str",
                "required": True
            }
        }
    },
    {
        "name": "product_database",
        "description": "A database that contains information about all the products of this company, including categories, prices, and stock levels.",
        "parameter_definitions": {
            "category": {
                "description": "Retrieves product information data for all products in this category.",
                "type": "str",
                "required": True
            }
        }
    }
]

Step 3: Create Custom Preamble (Optional)

We keep the same preamble as in the previous chapter.

preamble = """## Task & Context
You help people answer their questions and other requests interactively. You will be asked a very wide array of requests on all kinds of topics. You will be equipped with a wide range of search engines or similar tools to help you, which you use to research your answer. You should focus on serving the user's needs as best you can, which will be wide-ranging.

## Style Guide
Unless the user asks for a different style of answer, you should answer in full sentences, using proper grammar and spelling.
"""

Single-Step Tool Use

Let’s now take the code used in the previous chapter and automate the four-step tool use workflow. Let’s put the code in a function called run_assistant. Here’s what it does:

Gets the user message (Step 1)
Calls the Chat endpoint for tool call generation (Step 2)
If the response contains at least one tool call, executes the tool call(s) and gets the tool results (Step 3)
Generates the final response with citations (Step 4)

An overview of the four-step tool use workflow

The main difference from the previous chapter is that this time, we put Steps 2 and 3 in a loop, checking if the model’s response contains tool calls. This allows the assistant to handle any number of tool call steps (zero, one, or multiple) for a given user message.

The other key difference is we are also enabling the assistant to retain the state (or memory) of a conversation, which allows it to handle multi-turn chat scenarios. We do this by having the run_assistant function take the chat history from the previous turns as an argument and return the updated chat history.

model = "command-r-plus"

def run_assistant(message, chat_history=[]):
    # Step 1: Get user message
    print(f"Question:\n{message}")
    print("="*50)

    # Step 2: Generate tool calls (if any)    
    response = co.chat(
        message=message,
        model=model,
        preamble=preamble,
        tools=tools,
        chat_history=chat_history,
        force_single_step=True
    )

    while response.tool_calls:
        tool_calls = response.tool_calls
        
        if response.text:
            print("Tool plan:")
            print(response.text,"\n")
        print("Tool calls:")
        for call in tool_calls:
            print(f"Tool name: {call.name} | Parameters: {call.parameters}")
        print("="*50)
        
        # Step 3: Get tool results
        tool_results = []
        for tc in tool_calls:
            tool_call = {"name": tc.name, "parameters": tc.parameters}
            tool_output = functions_map[tc.name](**tc.parameters)
            tool_results.append({"call": tool_call, "outputs": [tool_output]})
        
        # Step 4: Generate response and citations                
        response = co.chat(
            message="",
            model=model,
            preamble=preamble,
            tools=tools,
            tool_results=tool_results,
            chat_history=response.chat_history
        )

        # Update the chat history
        chat_history = response.chat_history
        
    # Print final response
    print("Final response:")
    print(response.text)
    print("="*50)
    
    # Print citations (if any)
    if response.citations:
        print("Citations:")
        for citation in response.citations:
            print(citation)
        print("\nCited Documents:")
        for document in response.documents:
            print(document)
        print("="*50)
    
    return chat_history

Let’s now ask the assistant the first question, which is the same as in the previous chapter.

chat_history = run_assistant("Can you provide a sales summary for 29th September 2023?")

The assistant correctly identifies that, out of the two tools available, query_daily_sales_report is sufficient to answer the question. And it answers the question correctly.

Question:
Can you provide a sales summary for 29th September 2023?
==================================================
Tool calls:
Tool name: daily_sales_report | Parameters: {'day': '2023-09-29'}
==================================================
Final response:
On 29 September 2023, we made a total of 10,000 in sales from 250 units.
==================================================
Citations:
start=32 end=56 text='total of 10,000 in sales' document_ids=['daily_sales_report:0:2:0']
start=62 end=72 text='250 units.' document_ids=['daily_sales_report:0:2:0']

Cited Documents:
{'date': '2023-09-29', 'id': 'daily_sales_report:0:2:0', 'summary': 'Total Sales Amount: 10000, Total Units Sold: 250', 'tool_name': 'daily_sales_report'}
==================================================

In this example, the model requires a single step to answer the question, and within that single step, it requires just one tool call.

Single-Step, Parallel Tool Use

As mentioned, the model can decide that more than one tool is required to provide a response. This means calling multiple tools in parallel within the same step. This can either be:

Calling different tools in parallel
Calling the same tool multiple times in parallel
Or a combination of both

To illustrate this, let’s try to ask the model another question. Let’s ask about the sales summary of two different dates and also about the stock level information of a product category.

chat_history = run_assistant("Can you provide a sales summary for 28th and 29th September 2023 as well as the stock level of the products in the 'Electronics' category?")

This time, the model generates three tool calls in parallel:

Two queries to the sales database, one for each date
And one query to the product catalog

These are exactly what are needed to answer the question. And it correctly answers the question.

Question:
Can you provide a sales summary for 28th and 29th September 2023 as well as the stock level of the products in the 'Electronics' category?
==================================================
Tool calls:
Tool name: daily_sales_report | Parameters: {'day': '2023-09-28'}
Tool name: daily_sales_report | Parameters: {'day': '2023-09-29'}
Tool name: product_database | Parameters: {'category': 'Electronics'}
==================================================
Final response:
On 28 September 2023, the total sales amount was 5000, with 100 units sold. The following day, 29 September 2023, the total sales amount was 10000, with 250 units sold. 

Here is the stock level of the products in the 'Electronics' category:
- Smartphone (E1001) - 20
- Laptop (E1002) - 15
- Tablet (E1003) - 25
==================================================
Citations:
start=3 end=20 text='28 September 2023' document_ids=['daily_sales_report:0:2:0']
start=26 end=53 text='total sales amount was 5000' document_ids=['daily_sales_report:0:2:0']
start=60 end=74 text='100 units sold' document_ids=['daily_sales_report:0:2:0']
start=95 end=112 text='29 September 2023' document_ids=['daily_sales_report:1:2:0']
start=118 end=146 text='total sales amount was 10000' document_ids=['daily_sales_report:1:2:0']
start=153 end=167 text='250 units sold' document_ids=['daily_sales_report:1:2:0']
start=244 end=267 text='Smartphone (E1001) - 20' document_ids=['product_database:2:2:0']
start=270 end=289 text='Laptop (E1002) - 15' document_ids=['product_database:2:2:0']
start=292 end=311 text='Tablet (E1003) - 25' document_ids=['product_database:2:2:0']

Cited Documents:
{'date': '2023-09-28', 'id': 'daily_sales_report:0:2:0', 'summary': 'Total Sales Amount: 5000, Total Units Sold: 100', 'tool_name': 'daily_sales_report'}
{'date': '2023-09-29', 'id': 'daily_sales_report:1:2:0', 'summary': 'Total Sales Amount: 10000, Total Units Sold: 250', 'tool_name': 'daily_sales_report'}
{'category': 'Electronics', 'id': 'product_database:2:2:0', 'products': '[{"name":"Smartphone","price":500,"product_id":"E1001","stock_level":20},{"name":"Laptop","price":1000,"product_id":"E1002","stock_level":15},{"name":"Tablet","price":300,"product_id":"E1003","stock_level":25}]', 'tool_name': 'product_database'}
==================================================

Directly Answering

A key attribute of tool use systems is the model’s ability to choose the right tools for a task. This includes the ability to decide to not use any tool, and instead, respond to a user message directly.

This could happen if a user’s question inherently can be answered directly or none of the available tools are helpful to answer the question.

Comparing the sequence of events of using tools and directly answering

Let’s look at the first scenario. The question below asks about building a great company. It is a rather general question that a good LLM would be able to answer directly without needing any additional help.

chat_history = run_assistant("Give me 3 concise tips on how to build a great company")

And here’s the response. Notice that the model doesn’t trigger any tool calls, but instead goes directly to answering the question.

Question:
Give me 3 concise tips on how to build a great company
==================================================
Final response:
1. Start with a strong foundation: Build a solid business plan, secure adequate funding, and establish a clear mission and vision. 

2. Focus on people: Hire and retain talented employees by creating a positive company culture and offering competitive benefits. 

3. Stay agile: Adapt to market changes and be open to innovation. Stay connected to your customers' needs and wants and be willing to pivot when necessary.
==================================================

Let’s look at the other scenario. The question below asks about the company's employee count, which is a specific piece of information that an LLM would not possess and requires context from an external tool.

chat_history = run_assistant("How many employees does this company have?")

But because none of the available tools can provide this information, the model doesn’t attempt any tool call. Instead, it responds directly to the user question, mentioning that it doesn’t have the information needed to answer the question.

Question:
How many employees does this company have?
==================================================
Final response:
I don't have any information about the company's number of employees. Is there anything else I can help you with?
==================================================

State Management (Memory)

When we created the run_assistant function, we already enabled it to handle multi-turn chat scenarios. The function represents one turn of a conversation. For each turn, it accepts the latest chat history of a conversation and returns the updated one once the turn has been completed.

This can continue for any number of turns, the limit being the model’s maximum context length (with Command R/R+, that’s 128k tokens).

The chat history for each turn consists of:

The USER message
Followed by the CHATBOT message with the list of tool calls
Followed by the TOOL message with the list of tool results
Finally, followed by the CHATBOT message with the final response to the user

Maintaining the state over multiple chat turns (single-step tool use)

These four entries in the chat history directly map to the four steps in the tool use workflow. Here’s an example chat history taken from the first user message example earlier.

# Print chat history
for turn in chat_history:
    print(turn,"\n")


message='Can you provide a sales summary for 29th September 2023?' tool_calls=None role='USER' 

message=None tool_calls=[ToolCall(name='daily_sales_report', parameters={'day': '2023-09-29'})] role='CHATBOT' 

tool_results=[ToolResult(call=ToolCall(name='daily_sales_report', parameters={'day': '2023-09-29'}), outputs=[{'date': '2023-09-29', 'summary': 'Total Sales Amount: 10000, Total Units Sold: 250'}])] role='TOOL' 

message='On 29 September 2023, we sold 250 units, totalling a sales amount of 10,000.' tool_calls=None role='CHATBOT'

All this becomes part of the prompt to a chatbot at a particular turn, giving it the full context of a conversation.

Let’s try it out, starting with the same question about the sales summary.

chat_history = run_assistant("Can you provide a sales summary for 29th September 2023?")

This gives the same answer as before.

Question:
Can you provide a sales summary for 29th September 2023?
==================================================
Tool calls:
Tool name: daily_sales_report | Parameters: {'day': '2023-09-29'}
==================================================
Final response:
On 29 September 2023, the total sales amount was 10,000 and the total number of units sold was 250.
==================================================
Citations:
start=26 end=55 text='total sales amount was 10,000' document_ids=['daily_sales_report:0:2:0']
start=64 end=98 text='total number of units sold was 250' document_ids=['daily_sales_report:0:2:0']

Cited Documents:
{'date': '2023-09-29', 'id': 'daily_sales_report:0:2:0', 'summary': 'Total Sales Amount: 10000, Total Units Sold: 250', 'tool_name': 'daily_sales_report'}
==================================================

Let’s now ask a follow-up question, a rather vague question that a model would not be able to answer without the context of the previous turn. Here, we pass the first turn’s chat history to the run_assistant function.

chat_history = run_assistant("What about the 28th?", chat_history)

The model is able to infer that “the 28th” is likely referring to September 28th 2023, because of what was asked in the previous turn.

Question:
What about the 28th?
==================================================
Tool calls:
Tool name: daily_sales_report | Parameters: {'day': '2023-09-28'}
==================================================
Final response:
On 28 September 2023, the total sales amount was 5,000 and the total number of units sold was 100.
==================================================
Citations:
start=26 end=54 text='total sales amount was 5,000' document_ids=['daily_sales_report:0:6:0']
start=63 end=97 text='total number of units sold was 100' document_ids=['daily_sales_report:0:6:0']

Cited Documents:
{'date': '2023-09-28', 'id': 'daily_sales_report:0:6:0', 'summary': 'Total Sales Amount: 5000, Total Units Sold: 100', 'tool_name': 'daily_sales_report'}
==================================================

Let’s continue the conversation.

chat_history = run_assistant("How many units were sold over both days", chat_history)

Again, the model is able to infer what “both days” are from the chat context.

Question:
How many units were sold over both days
==================================================
Tool calls:
Tool name: daily_sales_report | Parameters: {'day': '2023-09-29'}
Tool name: daily_sales_report | Parameters: {'day': '2023-09-28'}
==================================================
Final response:
Combined, 28 and 29 September 2023 saw 350 units sold.
==================================================
Citations:
start=39 end=53 text='350 units sold' document_ids=['daily_sales_report:0:10:0', 'daily_sales_report:1:10:0']

Cited Documents:
{'date': '2023-09-29', 'id': 'daily_sales_report:0:10:0', 'summary': 'Total Sales Amount: 10000, Total Units Sold: 250', 'tool_name': 'daily_sales_report'}
{'date': '2023-09-28', 'id': 'daily_sales_report:1:10:0', 'summary': 'Total Sales Amount: 5000, Total Units Sold: 100', 'tool_name': 'daily_sales_report'}
==================================================

Here’s a look at the chat history. It consists of a set of USER, CHATBOT, and TOOL messages appended in the right sequence of turns, providing the model with the right context each time it generates a new response.

message='Can you provide a sales summary for 29th September 2023?' tool_calls=None role='USER' 

message=None tool_calls=[ToolCall(name='daily_sales_report', parameters={'day': '2023-09-29'})] role='CHATBOT' 

tool_results=[ToolResult(call=ToolCall(name='daily_sales_report', parameters={'day': '2023-09-29'}), outputs=[{'date': '2023-09-29', 'summary': 'Total Sales Amount: 10000, Total Units Sold: 250'}])] role='TOOL' 

message='On 29 September 2023, we sold 250 units, totalling a revenue of 10,000.' tool_calls=None role='CHATBOT' 

message='What about the 28th?' tool_calls=None role='USER' 

message=None tool_calls=[ToolCall(name='daily_sales_report', parameters={'day': '2023-09-28'})] role='CHATBOT' 

tool_results=[ToolResult(call=ToolCall(name='daily_sales_report', parameters={'day': '2023-09-28'}), outputs=[{'date': '2023-09-28', 'summary': 'Total Sales Amount: 5000, Total Units Sold: 100'}])] role='TOOL' 

message='On 28 September 2023, we sold 100 units, totalling a revenue of 5,000.' tool_calls=None role='CHATBOT' 

message='How many units were sold over both days' tool_calls=None role='USER' 

message=None tool_calls=[ToolCall(name='daily_sales_report', parameters={'day': '2023-09-29'}), ToolCall(name='daily_sales_report', parameters={'day': '2023-09-28'})] role='CHATBOT' 

tool_results=[ToolResult(call=ToolCall(name='daily_sales_report', parameters={'day': '2023-09-29'}), outputs=[{'date': '2023-09-29', 'summary': 'Total Sales Amount: 10000, Total Units Sold: 250'}]), ToolResult(call=ToolCall(name='daily_sales_report', parameters={'day': '2023-09-28'}), outputs=[{'date': '2023-09-28', 'summary': 'Total Sales Amount: 5000, Total Units Sold: 100'}])] role='TOOL' 

message='Over the two days, we sold 350 units.' tool_calls=None role='CHATBOT'

Here, the four entries in the chat history directly map to the four-step workflow. It starts with the USER message, the CHATBOT message with the list of tool calls, the TOOL message with the list of tool results, and the CHATBOT message with the final response to the user.

Conclusion

In this chapter, we added another tool to the structured data RAG assistant from the previous chapter. We also looked at the different dynamics this introduces, such as calling tools in parallel and being able to decide on the right tool(s) for a given user message.

However, all the examples so far have been confined to a single-step scenario. For example, “Create a plot of the number of employees at the three tech companies with the highest market cap in the United States in 2024.” requires a number of tool calls that happen in sequence. These types of tasks can’t be handled with single-step tool use.

An assistant's potential can only be truly unlocked when it can reason over complex tasks that require multiple steps of tool-calling. We’ll learn about exactly that in the next two chapters.

Let’s now turn our focus to multi-step tool use in Chapter 4.

Contents