We’ll use Cohere’s Python SDK for the code examples. Follow along in this notebook.

Introduction

Enterprises have a lot of data at their disposal, but much of it is hard to tap into. This is because a good chunk of this data is unstructured, and digging into it to find insights isn't straightforward. Traditional methods, like keyword matching, can only do so much and they often fail to capture the full context and depth of a document.

This is where we introduce the concept of text embeddings. Essentially, text embeddings are numerical representations created by language models that convert text into vectors. They capture and encode the context of a document. These vectors store a wealth of context about the documents they represent, opening up the possibility of various applications, from semantic search and retrieval-augmented generation (RAG) to topic modeling and text classification.

Cohere's Embed model, available on Amazon Bedrock, is a powerful text embeddings model that offers these capabilities. This model supports over 100 languages and is unique among text embedding models due to its emphasis on document quality for applications like semantic search.

For example, a global network security company wanted to build an AI assistant to help its IT professionals respond to queries on network security setup and security best practices. Its security policy documentation repository includes tens of thousands of articles consisting of unstructured data, and its legacy search tools often produce inaccurate results.

They built the AI assistant with retrieval-augmented generation (RAG). They used Cohere Embed for semantic search to generate embeddings of the security policy documentation and Cohere Rerank to further improve retrieval accuracy by selecting the most relevant documents for answer generation. The team then fine-tuned Cohere Command to optimize how it answered user questions.

Implementing an AI assistant at a global network security company

In this chapter, we'll explore how to use Cohere's Embed model on Amazon Bedrock. At the time of writing, users can choose between two models on Amazon Bedrock: Embed English and Embed Multilingual which covers 100+ languages. For the most updated list of models, visit the Amazon Bedrock website.

We also have a supplementary video on this topic for further reference:

0:00

/16:11

Code Walkthrough

Using Cohere Embed in a financial services scenario, let’s build an application that can perform semantic search across financial news in different languages.

Financial analysts need to digest a lot of content, such as financial publications and news media, to stay informed. A semantic search system helps analysts quickly search across numerous article titles in multiple languages for the most relevant articles, saving an enormous amount of time and effort. Unlike traditional keyword-based search, semantic search goes beyond matching exact terms and instead focuses on understanding the underlying concepts and relationships between words and phrases.

An overview of what we'll cover in the code walkthrough

Setup

First, let's install and import the necessary libraries and set up our Cohere client using the cohere SDK. To use Bedrock, we create a BedrockClient by passing the necessary AWS credentials.

# ! pip install cohere pandas hnswlib -q

import pandas as pd
import hnswlib
import re
import cohere
co = cohere.BedrockClient(
    aws_region="YOUR_AWS_REGION",
    aws_access_key="YOUR_AWS_ACCESS_KEY_ID",
    aws_secret_key="YOUR_AWS_SECRET_ACCESS_KEY",
    aws_session_token="YOUR_AWS_SESSION_TOKEN",
)

Download Dataset

We use a dataset (MultiFIN) containing a list of real-world article headlines covering 15 languages (English, Turkish, Danish, Spanish, Polish, Greek, Finnish, Hebrew, Japanese, Hungarian, Norwegian, Russian, Italian, Icelandic, and Swedish). This is an open-source dataset curated for financial natural language processing (NLP) and is available on a GitHub repository.

In our case, we’ve created a CSV file with MultiFIN’s data, as well as a column with translations. We don’t use this column to feed the model; we use it to help us follow along when we print the results for those who don’t speak Danish or Spanish. We point to that CSV to create our dataframe.

url = "https://raw.githubusercontent.com/cohere-ai/cohere-aws/main/notebooks/bedrock/multiFIN_train.csv"
df = pd.read_csv(url)

# Inspect dataset
df.head(5)

Pre-Process Dataset

MultiFIN has over 6,000 records in 15 different languages. For our example use case, we focus on three languages: English, Spanish, and Danish.

For this, we’ll need to do some pre-processing steps. First, we remove the duplicates, remove the languages other than the three we need, and pick the top 80 articles for demonstration purposes.

# Ensure there is no duplicated text in the headers
def remove_duplicates(text):
    return re.sub(r'((\b\w+\b.{1,2}\w+\b)+).+\1', r'\1', text, flags=re.I)

df ['text'] = df['text'].apply(remove_duplicates)

# Keep only selected languages
languages = ['English', 'Spanish', 'Danish']
df = df.loc[df['lang'].isin(languages)]

# Pick the top 80 longest articles
df['text_length'] = df['text'].str.len()
df.sort_values(by=['text_length'], ascending=False, inplace=True)
top_80_df = df[:80]

# Language distribution
top_80_df['lang'].value_counts()

Embed and Index Documents

Now, we want to embed our documents and store the embeddings. The embeddings are very large vectors that encapsulate the semantic meaning of our document. In particular, we use Cohere’s embed-multilingual-v3.0 model, which creates embeddings with 1,024 dimensions.

With the v3.0 embeddings models, we need to specify the input_type parameter to indicate the nature of the document. In semantic search applications, this is either search_document, which is for the documents to search, or search_query, which is for the search query that we’ll define later.

We also keep track of the language and translation of the document to enrich the display of the results.

Next, we create a search index using the hnsw vector library. This stores the embeddings in an index, which makes searching the documents more efficient.

# Embed documents
docs = top_80_df['text'].to_list()
docs_lang = top_80_df['lang'].to_list()
translated_docs = top_80_df['translation'].to_list() #for reference when returning non-English results
doc_embs = co.embed(texts=docs,
                    model="cohere.embed-multilingual-v3",
                    input_type='search_document').embeddings

# Create a search index
index = hnswlib.Index(space='ip', dim=1024)
index.init_index(max_elements=len(doc_embs), ef_construction=512, M=64)
index.add_items(doc_embs, list(range(len(doc_embs))))

Send Query and Retrieve Documents

Next, we build a function that takes a query as input, embeds it, and finds the four headers more closely related to it.

# Retrieval of 4 closest docs to query
def retrieval(query):
    # Embed query and retrieve results
    query_emb = co.embed(texts=[query],
                         model="cohere.embed-multilingual-v3",
                         input_type="search_query").embeddings
    doc_ids = index.knn_query(query_emb, k=3)[0][0] # we will retrieve 4 closest neighbors
    
    # Print and append results
    print(f"QUERY: {query.upper()} \n")
    retrieved_docs, translated_retrieved_docs = [], []
    
    for doc_id in doc_ids:
        # Append results
        retrieved_docs.append(docs[doc_id])
        translated_retrieved_docs.append(translated_docs[doc_id])
    
        # Print results
        print(f"ORIGINAL ({docs_lang[doc_id]}): {docs[doc_id]}")
        if docs_lang[doc_id] != "English":
            print(f"TRANSLATION: {translated_docs[doc_id]} \n----")
        else:
            print("----")
    print("END OF RESULTS \n\n")
    return retrieved_docs, translated_retrieved_docs

Let’s now try to query the index with a couple of examples, one each in English and Danish.

queries = [
    "Can data science help meet sustainability goals?", # English example
    "Hvor kan jeg finde den seneste danske boligplan?" # Danish example - "Where can I find the latest Danish property plan?"
]

for query in queries:
    retrieval(query)

Here’s the response for the English query, showing semantic search in action. Notice how the retrieval system was able to surface documents similar in meaning, i.e., data science vs. AI. Keyword-based search systems would not be able to capture this.

QUERY: CAN DATA SCIENCE HELP MEET SUSTAINABILITY GOALS? 

ORIGINAL (English): Using AI to better manage the environment could reduce greenhouse gas emissions, boost global GDP by up to 38m jobs by 2030
----
ORIGINAL (English): Quality of business reporting on the Sustainable Development Goals improves, but has a long way to go to meet and drive targets.
----
ORIGINAL (English): Only 10 years to achieve Sustainable Development Goals but businesses remain on starting blocks for integration and progress
----
END OF RESULTS

Here’s the response for the Danish query. This example highlights the ability to perform cross-lingual searches with the Embed Multilingual model. You can enter a query in one language and get relevant search results in other languages.

Another observation here is that the English acronym “PP&E” stands for “property, plant, and equipment,” and the model was able to connect it to the query.

QUERY: HVOR KAN JEG FINDE DEN SENESTE DANSKE BOLIGPLAN? 

ORIGINAL (Danish): Nyt fra CFOdirect: Ny PP&E-guide, FAQs om den nye leasingstandard, podcast om udfordringerne ved implementering af leasingstandarden og meget mere
TRANSLATION: New from CFOdirect: New PP&E guide, FAQs on the new leasing standard, podcast on the challenges of implementing the leasing standard and much more 
----
ORIGINAL (Danish): Lovforslag fremlagt om rentefri lån, udskudt frist for lønsumsafgift, førtidig udbetaling af skattekredit og loft på indestående på skattekontoen
TRANSLATION: Bills presented on interest -free loans, deferred deadline for payroll tax, early payment of tax credit and ceiling on the balance in the tax account 
----
ORIGINAL (Danish): Nyt fra CFOdirect: Shareholder-spørgsmål til ledelsen, SEC cybersikkerhedsguide, den amerikanske skattereform og meget mere
TRANSLATION: New from CFOdirect: Shareholder questions for management, the SEC cybersecurity guide, US tax reform and more 
----
END OF RESULTS

Conclusion

Semantic search applications, enabled by text embeddings, offer a significantly more effective approach to retrieving and analyzing information. Cohere's Embed model can do this across over 100 languages. Its application in fields like financial analysis, as demonstrated in this chapter, shows how it can transform data retrieval and processing tasks, saving time and improving accuracy.

In Chapter 5, we’ll switch to Amazon SageMaker and look at an example using the Cohere Rerank model on SageMaker.

Contents

Semantic Search Using Cohere Embed on Amazon Bedrock