Embed 3, our industry-leading AI search model, is now multimodal. It is capable of generating embeddings from both text and images.

This advancement enables enterprises to unlock real value from their vast amount of data stored in images. Businesses can now build systems that accurately and quickly search important multimodal assets such as complex reports, product catalogs, and design files to boost workforce productivity. Embed 3 is now the most generally capable multimodal embedding model on the market.

Find insights faster

Embedding models are an important tool within the modern enterprise AI toolkit. You can think of these models as the engine of intelligent search applications, helping employees and customers quickly locate specific information within massive libraries of data.

“Embed 3 is an essential part of our search and retrieval stack, ensuring we’re able to find relevant information and provide accurate answers in our end-user HR application. The fact that it works across 100+ languages helps us serve our global customer base.” - Willson Cross, Founder & CEO at Borderless AI

Behind the scenes, Embed 3 accomplishes this by translating input data into long strings of numbers which represent the meaning of the data. These numerical representations are then compared to each other to determine similarities and differences. Embed 3 places both text and image embeddings in the same vector space for an integrated experience.

This visual is simplified for illustrative purposes. In practice, the numerical representations of data (seen in the output column) are far longer and the vector space which stores them has a higher number of dimensions.

This similarity comparison enables applications to retrieve enterprise data which is relevant to an end-user query. In addition to being a fundamental component of semantic search systems, Embed 3 is useful in retrieval-augmented generation (RAG) systems as it ensures that generative models like our Command R series have relevant business context to inform their responses.

All businesses, across industry and size, can benefit from multimodal AI search. We have found customers to be particularly excited about real-world use cases, including:

Graphs & Charts: Visual representations are key to understanding complex data. Users can now effortlessly find the right diagrams to inform their business decisions. Simply describe a specific insight and Embed 3 will retrieve relevant graphs and charts, making data-driven decision making more efficient for employees across teams.
E-commerce Product Catalogs: Traditional search methods often limit customers to finding products through text-based product descriptions. Embed 3 transforms this search experience. Retailers can build applications which search product images in addition to text descriptions, creating a differentiated experience for shoppers and improving conversion rates.
Design Files & Templates: Designers often work with vast libraries of assets, relying on memory or rigorous naming conventions to organize visuals. Embed 3 makes it simple to locate specific UI mockups, visual templates, and presentation slides based on a text description. This streamlines the creative process.

At a time when businesses are increasingly expected to use their data to drive outcomes, Embed 3 offers a competitive advantage which accelerates productivity and improves customer experience.

Industry-leading accuracy and performance

Embed 3 is the most generally capable multimodal embedding model available, with state-of-the-art accuracy on various retrieval tasks for businesses and developers.

All text-to-image benchmarks are evaluated using Recall@10; text-to-text benchmarks are evaluated using NDCG@10. Text-to-text benchmark accuracy is based on BEIR, a dataset focused on out-of-domain retrievals (14 datasets). Generic text-to-image benchmark accuracy is based on Flickr and CoCo. eCommerce benchmark accuracy is based on a mix of product catalog and fashion catalog datasets. Design Files benchmark accuracy is based on a product design retrieval dataset constructed internally. Graphs & Charts benchmark accuracy is based on business reports and presentations constructed internally.

The text and image encoders within Embed 3 share a unified latent space. This enables users to include both image and text data within a single database, significantly reducing complexity compared to maintaining two separate databases. Our model is also uniquely performant at mixed modality searches. Other models tend to cluster text and image data into separate areas which leads to weak search results that are biased toward text-only data. Embed 3, on the other hand, prioritizes the meaning behind data, without biasing towards a specific modality, to ensure the most relevant results.

Retrieval accuracy is evaluated using NDCG@10. In this evaluation, we embed the Flickr image corpus alongside 500 irrelevant text chunks. We then compute NDCG@10 as normal using the Flickr queries and their corresponding images. CLIP’s low score is due to the fact that CLIP ranks irrelevant texts above relevant images.

In addition to leading accuracy and ease of use, Embed 3 continues to deliver the same useful enterprise search capabilities as before. For example, the model maintains strong performance on noisy real-world data and offers leading multilingual performance across 100+ languages.

Retrieval accuracy is evaluated using NDCG@10. In this evaluation, we embed images of infographics containing both text and visual components. Text within these infographics include German, Spanish, Portuguese, Vietnamese, Italian, Russian, Hungarian, French, Chinese, Korean, and Japanese. Queries are written in the same language as the target multimodal feature.

Availability and getting started

Embed 3 with new image search capabilities is available today on Cohere’s platform and on Amazon SageMaker. It is also available for private deployment into any Virtual Private Cloud (VPC) or on-premise environment.

To get started, please reach out to our sales team and find more technical details in our documentation.

Get Started!

Introducing Multimodal Embed 3: Powering AI Search