Contents

Introduction
Code Walkthrough
Setup
- Set Up the Tools
- Run the Tool Use Workflow
Single-Step Scenario
Multi-Step Scenario
Conclusion

We’ll use Cohere’s Python SDK for the code examples. Follow along in this notebook.

Introduction

Tool use extends the capabilities of retrieval-augmented generation (RAG) by making it possible to build applications that can not only answer questions but also automate tasks.

With a tool use approach, external systems are used to guide an LLM’s response, but they do so by leveraging a much bigger set of tools than what’s possible with RAG. The concept of tool use expands on LLMs' useful feature of being able to act as a reasoning and decision-making engine. Tool use also enables developers to build agentic applications that can take actions; that is, they are capable of doing both read and write operations on an external system.

For example, a global financial consultancy wanted to build an executive AI assistant for a national telecom company that could support executive decision-making. To meet the client’s requirements, the solution needed to extract information from internal document stores and real-time data sources.

Using Cohere’s tool use solution, along with RAG, the firm was able to leverage Command’s multi-step tool use capabilities and external agents (e.g., calculators and stock price sources) to retrieve and manipulate structured data.

Implementing a tool use workflow at a global financial consultancy

In this chapter, we'll explore how to build an agentic application that leverages tool using Cohere's Command R+ model on Amazon Bedrock. For the most updated list of Cohere models available on Amazon Bedrock, visit the Amazon Bedrock website.

Code Walkthrough

Let’s learn about tool use by building an onboarding assistant that helps new hires of a fictitious company called Co1t get set up in their first week. The assistant can help answer user questions about the company, search for information from emails, and create meeting appointments.

An overview of what we'll cover in the code walkthrough

Setup

First, let's install and import the necessary libraries and set up our Cohere client using the cohere SDK. To use Bedrock, we create a BedrockClient by passing the necessary AWS credentials.

# ! pip install cohere

import cohere

# Create Bedrock client via the native Cohere SDK
# Contact your AWS administrator for the credentials
co = cohere.BedrockClient(
    aws_region="YOUR_AWS_REGION",
    aws_access_key="YOUR_AWS_ACCESS_KEY_ID",
    aws_secret_key="YOUR_AWS_SECRET_ACCESS_KEY",
    aws_session_token="YOUR_AWS_SESSION_TOKEN",
)

Set Up the Tools

The pre-requisite, before we can run a tool use workflow, is to set up the tools. Let's create three tools:

search_faqs: A tool for searching the FAQs. For simplicity, we'll not implement any retrieval logic, but we'll simply pass a list of three predefined documents. In practice, we would set up a retrieval system as we did in Chapters 4, 5, and 6.
search_emails: A tool for searching the emails. Same as above, we'll simply pass a list of predefined emails.
create_calendar_event: A tool for creating new calendar events. Again, for simplicity, we'll only return mock successful event creations without actual implementation. In practice, we can connect to a calendar service API and implement all the necessary logic here.

Here, we are defining a Python function for each tool, but more broadly, the tool can be any function or service that can receive and send objects.

def search_faqs(query):
    faqs = [
    {"text" : "Submitting Travel Expenses:\nSubmit your expenses through our user-friendly finance tool."},
    {"text" : "Side Projects Policy:\nWe encourage you to explore your passions! Just ensure there's no conflict of interest with our business."},
    {"text" : "Wellness Benefits:\nTo promote a healthy lifestyle, we provide gym memberships, on-site yoga classes, and health insurance."}
    ]
    return  {"faqs" : faqs}

def search_emails(query):
    emails = [
    {"from": "hr@co1t.com", "to": "david@co1t.com", "date": "2024-06-24", "subject": "A Warm Welcome to Co1t, David!", "text": "We are delighted to have you on board. Please find attached your first week's agenda."},
    {"from": "it@co1t.com", "to": "david@co1t.com", "date": "2024-06-24", "subject": "Instructions for IT Setup", "text": "Welcome, David! To get you started, please follow the attached guide to set up your work accounts."},
    {"from": "john@co1t.com", "to": "david@co1t.com", "date": "2024-06-24", "subject": "First Week Check-In", "text": "Hi David, let's chat briefly tomorrow to discuss your first week. Also, come join us for lunch this Thursday at noon to meet everyone!"},
    ]
    return  {"emails" : emails}
    
def create_calendar_event(date: str, time: str, duration: int):
    # You can implement any logic here
    return {"is_success": True,
            "message": f"Created a {duration} hour long event at {time} on {date}"}
    
functions_map = {
    "search_faqs": search_faqs,
    "search_emails": search_emails,
    "create_calendar_event": create_calendar_event
}

The next step is to define the tool schemas in a format that can be passed to the Chat endpoint. The schema must contain the following fields: name, description, and parameter_definitions in the format shown below.

This schema informs the LLM about what the tool does, and the LLM decides whether to use a particular tool based on it. Therefore, the more descriptive and specific the schema, the more likely the LLM will make the right tool call decisions.

tools = [
    {
      "name": "search_faqs",
      "description": "Given a user query, searches a company's frequently asked questions (FAQs) list and returns the most relevant matches to the query.",
      "parameter_definitions": {
        "query": {
          "description": "The query from the user",
          "type": "str",
          "required": True
        }
      }
    }, 
    {
      "name": "search_emails",
      "description": "Given a user query, searches a person's emails and returns the most relevant matches to the query.",
      "parameter_definitions": {
        "query": {
          "description": "The query from the user",
          "type": "str",
          "required": True
        }
      }
    }, 
    {
      "name": "create_calendar_event",
      "description": "Creates a new calendar event of the specified duration at the specified time and date. A new event cannot be created on the same time as an existing event.",
      "parameter_definitions": {
        "date": {
          "description": "the date on which the event starts, formatted as mm/dd/yy",
          "type": "str",
          "required": True
        },
        "time": {
          "description": "the time of the event, formatted using 24h military time formatting",
          "type": "str",
          "required": True
        },
        "duration": {
          "description": "the number of hours the event lasts for",
          "type": "float",
          "required": True
        }
      }
    }
]

Run the Tool Use Workflow

The tool is now ready to use. We can think of a tool use system as consisting of four components:

The user
The application
The LLM
The tools

At its most basic, these four components interact in a workflow through four steps:

Step 1: Get user message. The LLM gets the user message (via the application).
Step 2: Generate tool calls. The LLM makes a decision on the tools to call (if any) and generates the tool calls.
Step 3: Get tool results. The application executes the tools and sends the tool results to the LLM.
Step 4: Generate response and citations. The LLM generates the response and citations and sends them back to the user.

Let's create a function called run_assistant to implement these steps and print out the key events and messages along the way. This function also optionally accepts the chat history as an argument to keep the state in a multi-turn conversation.

model = "cohere.command-r-plus-v1:0"

preamble="""## Task and Context
You are an assistant who assist new employees of Co1t with their first week. You respond to their questions and assist them with their needs. Today is Monday, June 24, 2024"""

def run_assistant(message, chat_history=None):
    
    if chat_history is None:
        chat_history = []
    
    # Step 1: get user message
    print(f"Question:\n{message}")
    print("="*50)

    # Step 2: Generate tool calls (if any)    
    response = co.chat(
        message=message,
        model=model,
        preamble=preamble,
        tools=tools,
        chat_history=chat_history
    )

    while response.tool_calls:
        tool_calls = response.tool_calls
        
        if response.text:
            print("Tool plan:")
            print(response.text,"\n")
        print("Tool calls:")
        for call in tool_calls:
            print(f"Tool name: {call.name} | Parameters: {call.parameters}")
        print("="*50)
        
        # Step 3: Get tool results
        tool_results = []
        for tc in tool_calls:
            tool_call = {"name": tc.name, "parameters": tc.parameters}
            tool_output = functions_map[tc.name](**tc.parameters)
            tool_results.append({"call": tool_call, "outputs": [tool_output]})
        
        # Step 4: Generate response and citations                
        response = co.chat(
            message="",
            model=model,
            preamble=preamble,
            tools=tools,
            tool_results=tool_results,
            chat_history=response.chat_history
        )

        chat_history = response.chat_history
        
    # Print final response
    print("RESPONSE:\n")
    print(response.text)
    print("="*50)
    
    # Print citations (if any)
    if response.citations:
        print("\nCITATIONS:\n")
        for citation in response.citations:
            print(citation)

        print("\nCITED REFERENCES:\n")
        for document in response.documents:
            print(document)
                
    return chat_history

Single-Step Scenario

Cohere supports running tool use in single-step and multi-step modes. In a single-step scenario, the model will make only one round of tool calling. Having said that, within this one round, the model can decide to call multiple tools in parallel. This can be calling the same tool multiple times, calling different tools, or a combination of both.

Let’s look at an example of a new hire asking about IT access and the travel expense process.

chat_history = run_assistant("Any emails about setting up IT access? Also, how do I submit travel expenses?")

Given three tools to choose from, the model is able to pick the right tools (in this case, search_faqs and search_emails) based on what the user is asking for.

Also, notice that the model first generates a plan about what it should do ("I will do ...") before actually generating the tool call(s).

Additionally, the model also generates fine-grained citations in tool use mode based on the tool results it receives, the same way we saw with RAG.

Question:
Any emails about setting up IT access? Also, how do I submit travel expenses?
==================================================
Tool plan:
I will search for emails about setting up IT access and travel expenses. 

Tool calls:
Tool name: search_emails | Parameters: {'query': 'IT access'}
Tool name: search_emails | Parameters: {'query': 'travel expenses'}
==================================================
RESPONSE:

You have an email from *it@co1t.com* with the subject "Instructions for IT Setup". It contains an attached guide to set up your work accounts. 

To submit travel expenses, you can send an email to *expenses@co1t.com*.
==================================================

CITATIONS:

start=23 end=35 text='*it@co1t.com' document_ids=['search_emails:0:2:0', 'search_emails:1:2:0']
start=54 end=80 text='"Instructions for IT Setup' document_ids=['search_emails:0:2:0', 'search_emails:1:2:0']
start=98 end=141 text='attached guide to set up your work accounts' document_ids=['search_emails:0:2:0', 'search_emails:1:2:0']

CITED REFERENCES:

{'emails': '[{"date":"2024-06-24","from":"hr@co1t.com","subject":"A Warm Welcome to Co1t, David!","text":"We are delighted to have you on board. Please find attached your first week\'s agenda.","to":"david@co1t.com"},{"date":"2024-06-24","from":"it@co1t.com","subject":"Instructions for IT Setup","text":"Welcome, David! To get you started, please follow the attached guide to set up your work accounts.","to":"david@co1t.com"},{"date":"2024-06-24","from":"john@co1t.com","subject":"First Week Check-In","text":"Hi David, let\'s chat briefly tomorrow to discuss your first week. Also, come join us for lunch this Thursday at noon to meet everyone!","to":"david@co1t.com"}]', 'id': 'search_emails:0:2:0', 'tool_name': 'search_emails'}
{'emails': '[{"date":"2024-06-24","from":"hr@co1t.com","subject":"A Warm Welcome to Co1t, David!","text":"We are delighted to have you on board. Please find attached your first week\'s agenda.","to":"david@co1t.com"},{"date":"2024-06-24","from":"it@co1t.com","subject":"Instructions for IT Setup","text":"Welcome, David! To get you started, please follow the attached guide to set up your work accounts.","to":"david@co1t.com"},{"date":"2024-06-24","from":"john@co1t.com","subject":"First Week Check-In","text":"Hi David, let\'s chat briefly tomorrow to discuss your first week. Also, come join us for lunch this Thursday at noon to meet everyone!","to":"david@co1t.com"}]', 'id': 'search_emails:1:2:0', 'tool_name': 'search_emails'}

Multi-Step Scenario

Cohere supports running more complex tasks in tool use — tasks that require tool calls to happen in a sequence. This is referred to as "multi-step" tool use.

To illustrate this, let's ask the assistant to block time for any lunch invites received in the email.

chat_history = run_assistant("Can you check if there are any lunch invites, and for those days, block an hour on my calendar from 12-1PM.")

Here, we see the assistant running these steps:

First, it calls the search_emails tool to find any lunch invites, and it finds one.
Next, it calls the create_calendar_event tool to create an event to block the person's calendar on the day mentioned in the email.

This is also an example of tool use enabling a write operation instead of just a read operation that we saw with RAG.

Question:
Can you check if there are any lunch invites, and for those days, block an hour on my calendar from 12-1PM.
==================================================
Tool plan:
I will search the user's emails for lunch invites, and then create calendar events for the dates of the invites. 

Tool calls:
Tool name: search_emails | Parameters: {'query': 'lunch invite'}
==================================================
Tool plan:
I have found an email from John inviting David for lunch on Thursday at noon. I will now create a calendar event for this lunch. 

Tool calls:
Tool name: create_calendar_event | Parameters: {'date': '06/27/24', 'duration': 1, 'time': '12:00'}
==================================================
RESPONSE:

I have found an email from John inviting you for lunch on Thursday at noon. I have created a calendar event for this lunch.
==================================================

CITATIONS:

start=16 end=31 text='email from John' document_ids=['search_emails:0:2:0']
start=49 end=74 text='lunch on Thursday at noon' document_ids=['search_emails:0:2:0']
start=93 end=107 text='calendar event' document_ids=['create_calendar_event:0:4:0']

CITED REFERENCES:

{'emails': '[{"date":"2024-06-24","from":"hr@co1t.com","subject":"A Warm Welcome to Co1t, David!","text":"We are delighted to have you on board. Please find attached your first week\'s agenda.","to":"david@co1t.com"},{"date":"2024-06-24","from":"it@co1t.com","subject":"Instructions for IT Setup","text":"Welcome, David! To get you started, please follow the attached guide to set up your work accounts.","to":"david@co1t.com"},{"date":"2024-06-24","from":"john@co1t.com","subject":"First Week Check-In","text":"Hi David, let\'s chat briefly tomorrow to discuss your first week. Also, come join us for lunch this Thursday at noon to meet everyone!","to":"david@co1t.com"}]', 'id': 'search_emails:0:2:0', 'tool_name': 'search_emails'}
{'id': 'create_calendar_event:0:4:0', 'is_success': 'true', 'message': 'Created a 1 hour long event at 12:00 on 06/27/24', 'tool_name': 'create_calendar_event'}

Conclusion

In this chapter, we explored how to use Command R+ on Amazon Bedrock in an application that implements tool use. We covered the steps to create and define the tools and looked at how tool use works in single-step and multi-step scenarios.

If you want to learn more LLM use cases, visit our LLMU chapter on use case patterns.

In Chapter 8, we’ll look at an example of fine-tuning the Command R model using SageMaker.

Tool Use and Agents on Amazon Bedrock