We’ll use Cohere’s Python SDK for the code examples. Follow along in this notebook.

Introduction
ReAct-Style Prompting
Multi-Step Tool Use Workflow
- Planning and Execution
- Self-Correction
Setting Up the Tools
Multi-Step Tool Use
Tool Planning
Multi-Step, Parallel Tool Use
State Management (Memory)
Conclusion

Introduction

In Chapter 2, we briefly discussed that multi-step tool use happens when more than one tool is required and the output of one tool is needed as the input to another tool.

In other words, tool-calling needs to happen in a sequence.

This is especially true in complex workflows where tasks have to be broken down into a sequence of steps.

Let's say a user asks, "Plot Apple's 2023 revenue on a bar chart".

This requires first a web search for Apple's revenue and then generating the right Python code using the search results. In the same way as a single-step scenario, multi-step tool use can also happen in parallel.

Multi-step tool use serves tasks where tool-calling needs to happen in a sequence

In this chapter, we’ll focus on multi-step tool use and formalize our understanding through code examples.

We’ll build a calendar assistant that can check a user’s calendar and create new appointments based on available times. This use case requires tool calls to happen in sequence, i.e., multi-step. It is also an example of using tools for performing not just a read operation (retrieving information) but also a write operation (taking actions).

ReAct-Style Prompting

A common approach for multi-step tool use implementation is using the ReAct-style of prompting.

The ReAct framework (Yao et al., 2022) is a technique for improving the accuracy and reliability of responses from LLMs. It combines "reasoning" and "acting," inspired by the natural human ability to learn and make decisions by blending these two elements. It allows an LLM to create a plan to solve a task and then execute the plan step by step in an iterative process, where it reflects and adjusts its plan after each action it takes. By integrating both reasoning and acting within LLMs, ReAct guides them to produce verbal reasoning paths and specific actions for a given task.

The framework follows a series of Thought-Action-Observation reasoning sequences. Given a user message, the LLM first generates a thought, which includes the plan of what it needs to do. Then, it generates an action, which is a call to an external tool. The result of this call is then passed to its observation step. It will repeat this sequence until a task has been fulfilled.

Multi-Step Tool Use Workflow

Cohere’s multi-step tool workflow uses a style similar to the ReAct framework. The prompt is comprised of:

The preamble, consisting of the system and custom (if defined) preambles
The list of available tools, taken from the tool schema defined by the developer
The user message
And finally, the instructions to run the ReAct-style workflow (to be described next)

Planning and Execution

You can view the complete prompt to the model by enabling return_prompt to True when calling the Chat endpoint. But for our purpose, let’s just focus on the portion with the instructions to run the ReAct-style workflow.

...
Carefully perform the following instructions, in order, starting each with a new line.
Firstly, You may need to use complex and advanced reasoning to complete your task and answer the question. Think about how you can use the provided tools to answer the question and come up with a high level plan you will execute.
Write 'Plan:' followed by an initial high level plan of how you will solve the problem including the tools and steps required.
Secondly, Carry out your plan by repeatedly using actions, reasoning over the results, and re-evaluating your plan. Perform Action, Observation, Reflection steps with the following format. Write 'Action:' followed by a json formatted action containing the "tool_name" and "parameters"
 Next you will analyze the 'Observation:', this is the result of the action.
After that you should always think about what to do next. Write 'Reflection:' followed by what you've figured out so far, any changes you need to make to your plan, and what you will do next including if you know the answer to the question.
... (this Action/Observation/Reflection can repeat N times)
Thirdly, Decide which of the retrieved documents are relevant to the user's last input by writing 'Relevant Documents:' followed by comma-separated list of document numbers. If none are relevant, you should instead write 'None'.
Fourthly, Decide which of the retrieved documents contain facts that should be cited in a good answer to the user's last input by writing 'Cited Documents:' followed a comma-separated list of document numbers. If you dont want to cite any of them, you should instead write 'None'.
Fifthly, Write 'Answer:' followed by a response to the user's last input in high quality natural english. Use the retrieved documents to help you. Do not insert any citations or grounding markup.
Finally, Write 'Grounded answer:' followed by a response to the user's last input in high quality natural english.
...

The first instruction is to perform a “planning” step consisting of an initial high-level plan of how the model will solve the problem, including the tools and steps required.

This is followed by actually carrying out the plan by repeatedly using actions (tool calls), with every step going through an Action-Observation-Reflection sequence.

With the Cohere API, this sequence of reasoning is captured as an assistant’s tool plan before generating the actual tool calls, as we’ll see in the code example.

Cohere’s multi-step tool use workflow uses a style similar to the ReAct framework

Self-Correction

What’s not explicitly mentioned in the prompt but something the model can handle is self-correction.

So far, we have assumed that every tool call will return a successful result. But that does not necessarily have to be the case. Calling external functions could instead return errors, such as missing information, unsuccessful attempts, and many others.

For example, when calling a web search as a tool call, a timeout error might occur for some reason. That means that the tool result returned will not contain the required information to generate a response.

In this kind of situation, the model will be able to reflect on this returned result and take a different course of action than what was originally planned. In this case, instead of immediately going to the next task it had planned, it might attempt to rerun the same step that returned the error.

And with that, let’s start building our calendar assistant.

Setting Up the Tools

Let’s first set up the Cohere client. We’ll also need to import the json module for use later.

# pip install cohere

import json
import cohere
co = cohere.Client(api_key="YOUR_COHERE_API_KEY") # Get your API key: https://dashboard.cohere.com/api-keys

Step 1: Create Tools

Let’s now create two tools:

A function called list_calendar_events to list the existing calendar events based on a given date. For simplicity, we are not going to connect to an actual calendar. Instead, we’ll include a mock events record, which is a simple list of events in a day.
A function called create_calendar_event to create a new calendar event based on the provided date, time, and duration. Again, to keep things simple, we will not make actual changes to a database but instead will just return a simple success message for illustration purposes.

def list_calendar_events(date: str):
  events = [{"start": "8:00", "end": "8:59"}, {"start": "9:00", "end": "9:59"}, {"start": "11:00", "end": "11:59"},{"start": "12:00", "end": "12:59"}]

  return {
        "existing_events": events
    }

def create_calendar_event(date: str, time: str, duration: int):
  
  return {
        "is_success": True,
        "message": f"Created a {duration} hour long event at {time} on {date}"
    }

functions_map = {
    "list_calendar_events": list_calendar_events,
    "create_calendar_event": create_calendar_event
}

Step 2: Define Tool Schema

Next, we define the tool schema for the two tools.

tools = [
    {
      "name": "list_calendar_events",
      "description": "Returns a list of existing events for the specified date, including the start time and end time for each event.",
      "parameter_definitions": {
        "date": {
          "description": "the date to list events for, formatted as mm/dd/yy",
          "type": "str",
          "required": True
        }
      }
    }, 
    {
      "name": "create_calendar_event",
      "description": "Creates a new calendar event of the specified duration at the specified time and date. A new event cannot be created on the same time as an existing event.",
      "parameter_definitions": {
        "date": {
          "description": "the date on which the event starts, formatted as mm/dd/yy",
          "type": "str",
          "required": True
        },
        "time": {
          "description": "the time of the event, formatted using 24h military time formatting",
          "type": "str",
          "required": True
        },
        "duration": {
          "description": "the number of hours the event lasts for",
          "type": "float",
          "required": True
        }
      }
    }
]

Step 3: Create Custom Preamble (Optional)

Let’s also create a custom preamble. The important part here is specifying the date since we are not implementing any date-handling with the list_calendar_events tool. What we’re doing is we are nudging the model to specify this date in the create_calendar_event tool call, which requires a date field.

preamble="""## Task & Context
You are a calendar assistant who helps people schedule events on their calendar. You must make sure that a new event does not overlap with any existing event.
Today is Thursday, May 23, 2024
"""

Multi-Step Tool Use

Let’s reuse the run_assistant function we created in the previous chapter. Here’s a recap of what it does.

Get the user message (Step 1)
Call the Chat endpoint for tool call generation (Step 2)
If the response contains at least one tool call, execute the tool call(s) and get the tool results (Step 3)
Repeat Steps 2 and 3 until there are no more tool calls
Generate the final response with citations (Step 4)

An overview of the four-step tool use workflow

The good news is that it is already compatible with multi-step tool use because we have already implemented Steps 2 and 3 to run in a loop to handle zero, one, or multiple steps of tool calls.

But there is one change we need to make. Here, we remove the force_single_step argument as it is False by default. This tells the API to enable a multi-step workflow and use the right prompt template to run the workflow.

model = "command-r-plus"

def run_assistant(message, chat_history=[]):
    # Step 1: Get user message
    print(f"Question:\n{message}")
    print("="*50)

    # Step 2: Generate tool calls (if any)    
    response = co.chat(
        message=message,
        model=model,
        preamble=preamble,
        tools=tools,
        chat_history=chat_history
    )

    while response.tool_calls:
        tool_calls = response.tool_calls
        
        if response.text:
            print("Tool plan:")
            print(response.text,"\n")
        print("Tool calls:")
        for call in tool_calls:
            print(f"Tool name: {call.name} | Parameters: {call.parameters}")
        print("="*50)
        
        # Step 3: Get tool results
        tool_results = []
        for tc in tool_calls:
            tool_call = {"name": tc.name, "parameters": tc.parameters}
            tool_output = functions_map[tc.name](**tc.parameters)
            tool_results.append({"call": tool_call, "outputs": [tool_output]})
        
        # Step 4: Generate response and citations                
        response = co.chat(
            message="",
            model=model,
            preamble=preamble,
            tools=tools,
            tool_results=tool_results,
            chat_history=response.chat_history
        )

        # Update the chat history
        chat_history = response.chat_history
        
    # Print final response
    print("Final response:")
    print(response.text)
    print("="*50)
    
    # Print citations (if any)
    if response.citations:
        print("Citations:")
        for citation in response.citations:
            print(citation)
        print("\nCited Documents:")
        for document in response.documents:
            print(document)
        print("="*50)
    
    return chat_history

Let’s now ask the assistant the first question, starting with a simple one about the number of meetings for the day. This requires just a single step of tool calling to check existing calendar events.

chat_history = run_assistant("How many meetings do I have today")

That’s exactly what the assistant does, and it gives the correct answer.

Question:
How many meetings do I have today
==================================================
Tool plan:
I will use the 'list_calendar_events' tool to find out how many meetings the user has today. 

Tool calls:
Tool name: list_calendar_events | Parameters: {'date': '05/23/2024'}
==================================================
Final response:
You have four meetings today.
==================================================
Citations:
start=9 end=22 text='four meetings' document_ids=['list_calendar_events:0:2:0']

Cited Documents:
{'existing_events': '[{"end":"8:59","start":"8:00"},{"end":"9:59","start":"9:00"},{"end":"11:59","start":"11:00"},{"end":"12:59","start":"12:00"}]', 'id': 'list_calendar_events:0:2:0', 'tool_name': 'list_calendar_events'}
==================================================

Let’s now try to ask a question that requires multi-step tool calling — this one asking the assistant to book an appointment. To complete this task, the assistant will have to first query the existing events and then use the information to create a new event that wouldn’t cause a conflict.

chat_history = run_assistant("Create an hour-long appointment for the first available free slot after 9am")

Here’s the assistant’s response. In the first step, it calls the list_calendar_events tool to get a list of existing events.

In the second step, it calls the create_calendar_event tool to create a new event. It creates a new event at 10 a.m., which is indeed the first available slot after 9 a.m.

Question:
Create an hour-long appointment for the first available free slot after 9am
==================================================
Tool plan:
I will first check the user's calendar for 23/05/2024 to see if there are any free slots after 9am. I will then create an hour-long appointment for the first available free slot. 

Tool calls:
Tool name: list_calendar_events | Parameters: {'date': '05/23/2024'}
==================================================
Tool plan:
The user's calendar shows that they have a free slot from 10:00 to 10:59. I will now create an hour-long appointment for this time slot. 

Tool calls:
Tool name: create_calendar_event | Parameters: {'date': '05/23/2024', 'duration': 1, 'time': '10:00'}
==================================================
Final response:
I've created a 1-hour long appointment for 10:00 on 23/05/2024.
==================================================
Citations:
start=15 end=62 text='1-hour long appointment for 10:00 on 23/05/2024' document_ids=['create_calendar_event:0:4:0']

Cited Documents:
{'id': 'create_calendar_event:0:4:0', 'is_success': 'true', 'message': 'Created a 1 hour long event at 10:00 on 05/23/2024', 'tool_name': 'create_calendar_event'}
==================================================

And here’s a look at the chat history. Compared to the single-step scenario, it contains multiple pairs of CHATBOT - TOOL messages, with each pair being one step in the sequence of tool calls. In this particular case, we have two of them.

message='Create an hour-long appointment for the first available free slot after 9am' tool_calls=None role='USER' 

message="I will first check the user's calendar for 23/05/2024 to see when they are free after 9am. I will then create an hour-long appointment for the first available slot." tool_calls=[ToolCall(name='list_calendar_events', parameters={'date': '05/23/2024'})] role='CHATBOT' 

tool_results=[ToolResult(call=ToolCall(name='list_calendar_events', parameters={'date': '05/23/2024'}), outputs=[{'existing_events': [{'end': '8:59', 'start': '8:00'}, {'end': '9:59', 'start': '9:00'}, {'end': '11:59', 'start': '11:00'}, {'end': '12:59', 'start': '12:00'}]}])] role='TOOL' 

message='The user has an appointment from 9am to 9:59am. I will now create an hour-long appointment for the first available slot after this.' tool_calls=[ToolCall(name='create_calendar_event', parameters={'date': '05/23/2024', 'duration': 1, 'time': '10:00'})] role='CHATBOT' 

tool_results=[ToolResult(call=ToolCall(name='create_calendar_event', parameters={'date': '05/23/2024', 'duration': 1, 'time': '10:00'}), outputs=[{'is_success': True, 'message': 'Created a 1 hour long event at 10:00 on 05/23/2024'}])] role='TOOL' 

message="I've created an hour-long appointment for 10am on 23/05/2024." tool_calls=None role='CHATBOT'

Comparing the events within a single chat turn for single-step and multi-step scenarios

Tool Planning

The chat history also highlights another difference between the single-step and multi-step scenarios.

Notice that in the multi-step example above, the CHATBOT turns generate a textual response at each tool-calling step (look for the message) before making the actual tool calls.

Let’s call this the model’s tool plan. These intermediate responses are the model’s internal reasoning logic, which guides its next course of action.

This goes back to the ReAct framework discussed earlier. The first of these plans is the assistant’s initial planning step, which lists its set of actions to take. The subsequent responses consist of its reflection on the current state and decision on the next immediate actions.

Here’s how the tool plan in multi-step looks again in the chat history.

...

message='The user has an appointment from 9am to 9:59am. I will now create an hour-long appointment for the first available slot after this.' tool_calls=[ToolCall(name='create_calendar_event', parameters={'date': '05/23/2024', 'duration': 1, 'time': '10:00'})] role='CHATBOT' 

...

Meanwhile, in the single-step case, the model doesn’t generate any tool plan (message=None) but instead generates the tool calls directly.

...

message=None tool_calls=[ToolCall(name='daily_sales_report', parameters={'day': '2023-09-29'})] role='CHATBOT' 
...

Multi-step tool use contains tool plans, which single-step doesn't

Multi-Step, Parallel Tool Use

Let’s now look at an example of multi-step, parallel tool use. Let’s ask the assistant to create two separate events on the same day.

chat_history = run_assistant("Book two hour-long appointments for any available time between 8am to 6pm")

And here’s the assistant’s response. It follows the same two steps as in the earlier example: first, checking the existing events, and second, creating the new events.

The difference this time is that since it needs to create two events, it calls create_calendar_event twice within the same step. Here, the assistant makes the correct judgment and tool calls are independent of each other, which means they can be done in parallel.

Question:
Create two hour-long appointments for any available time between 8am to 6pm
==================================================
Tool plan:
I will first list the events for 23/05/2024. Then, I will create two hour-long appointments for any available time between 8am and 6pm. 

Tool calls:
Tool name: list_calendar_events | Parameters: {'date': '05/23/2024'}
==================================================
Tool plan:
The available times between 8am and 6pm are: 10am-11am, 1pm-6pm. I will now create two hour-long appointments for the user during these times. 

Tool calls:
Tool name: create_calendar_event | Parameters: {'date': '05/23/2024', 'duration': 1, 'time': '10:00'}
Tool name: create_calendar_event | Parameters: {'date': '05/23/2024', 'duration': 1, 'time': '13:00'}
==================================================
Final response:
I have created two hour-long appointments for you today: one at 10am and another at 1pm.
==================================================
Citations:
start=64 end=68 text='10am' document_ids=['create_calendar_event:0:4:0']
start=84 end=87 text='1pm' document_ids=['create_calendar_event:1:4:0']

Cited Documents:
{'id': 'create_calendar_event:0:4:0', 'is_success': 'true', 'message': 'Created a 1 hour long event at 10:00 on 05/23/2024', 'tool_name': 'create_calendar_event'}
{'id': 'create_calendar_event:1:4:0', 'is_success': 'true', 'message': 'Created a 1 hour long event at 13:00 on 05/23/2024', 'tool_name': 'create_calendar_event'}
==================================================

State Management (Memory)

In the previous chapter, we saw how state management works with the Chat endpoint from the context of single-step tool use.

Multi-step tool use works the same way. The chat history for each turn consists of the following messages, and will accumulate with every new turn.

The USER message
Followed by the CHATBOT message with the list of tool calls
Followed the TOOL message with the list of tool results
Finally, followed by the CHATBOT message with the final response to the user

The difference here is, as we’ve seen, that each turn can have multiple pairs of CHATBOT - TOOL messages, representing its multi-step nature. The other difference is the presence of the tool plans, which are not present in the single-step case.

Maintaining the state over multiple chat turns (multi-step tool use)

Conclusion

In this chapter, we learned about multi-step tool use by creating a calendar assistant that can check and create events on behalf of a user. We saw how the multi-step approach enables agents to perform a complex task via repeated sequences of planning, action, observation, and reflection. We also saw how all this can be done using the Cohere native API.

There is another way you can run multi-step tool use with Cohere, and that is via open-source frameworks. We have worked with the Langchain team to make it simple for developers to build Cohere-backed agents on Langchain.

In Chapter 5, we’ll learn how to use Langchain to create a multi-step agent.

Contents