As more businesses adopt generative AI, they’re realizing both the huge potential and the challenges of unlocking the value of their data.

Organizations hold vast amounts of data that can provide value to their employees and customers through advanced AI search, retrieval, and generation. But that data isn’t just held in simple text documents or tables; much of the information is found in different formats such as images, audio, and even video that are more complicated to process.

Until recently, integrating those data types with advanced AI systems like retrieval-augmented generation (RAG) has been hard to do at scale. Text embeddings – which capture the meaning of data in numerical representations – have more commonly been used to process text-based documents.

That’s now changing with recent advancements in multimodal embeddings. As the term suggests, this will allow organizations to feed different “modes” of data into their system’s embedding model, which can extract the meaning of the information regardless of the format.

In this article, we’ll address why multimodal embeddings are critical for enterprise AI and how to effectively use them with RAG systems.

Why are multimodal embeddings important?

Multimodal embeddings promise to give organizations a wider, more unified view of their data, whether that is from complex reports, slideshow files such as Powerpoint, product catalogs, graphs, or images such as written notes. In turn, that will enable them to draw richer insights and analyses from their data and to offer more sophisticated AI-enabled features to their customers and employees.

Users can retrieve relevant graphs and charts based on queries for a particular insight or trend, for example. Companies can build applications that enable customers to search schematics based on visual similarities without requiring text descriptions.

Retailers could use multimodal embeddings to integrate the large volume of visual, textual, and pricing data they have to offer more personalized recommendations to customers. A multimodal system might notice that a shopper prefers certain visual designs and price points based on their browsing and offer them similar items, for example.

Many businesses, including consultancy firms, have troves of valuable knowledge and insights stored in PowerPoint presentations, which are a mix of text, images, and graphs. Multimodal embedding could enable them to retrieve the whole spectrum of insights and data needed to create custom proposals.

Or consider a healthcare company seeking to integrate written medical notes and images, such as lab results and MRI scans, into its GenAI system. Multimodal embeddings could help doctors classify MRI images into disease subtypes and phenotypical information (e.g., neuropsychological traits), resulting in a more efficient and comprehensive diagnostic process. It could also find patterns across different data types, giving medical professionals deeper insights into disease symptoms and treatments.

How does it work?

Just as with text-only embedding, multimodal embedding works by encoding data as vectors based on its meaning. Related data types (whether they originate from text, images or other formats) are stored in the same latent space in the vector database.

Once stored as a vector, the original format of the data becomes irrelevant – the database is simply made up of numbers that represent meaning. Each vector is linked to metadata that 'points back' to its original source, enabling retrieval or reference to the original data when necessary.

A user’s query will trigger a so-called similarity search which finds the closest matches based on the vector positions. If the data type is an image, for example, it is first deconstructed into its essential components. A neural network then attributes more detail, nuance, and complexity to the image in order to encode it accurately based on its meaning.

Say the image was of a cat, for example. The network would first identify segments of the image as being “cat-like,” such as its triangular ears, fur patterns, and tail. It would then progress through deeper layers of cat identity markers, such as fur texture, whiskers, breed, and its setting.

Its vector position in the embedding model would be encoded with all of this information and be located in latent space close to data with similar meaning, regardless of their original format.

With text-only embedding, a query seeking information about cats might generate results showing a British Shorthair or other breeds. With multimodal embedding, though, it might also show an image of a lion or of the cursive word “panther.” In effect, multimodal embedding means you are leaving far fewer stones unturned in data retrieval.

What are some challenges of implementing multimodal embeddings?

An important consideration for organizations is that multimodal embedding models integrate seamlessly with existing vectorized data.

Cohere recently made Embed 3 multimodal, placing both text and image embeddings in the same vector space for an integrated experience. This enables organizations to upgrade to multimodal functionality without the expense and hassle of re-vectorizing all of their text data. Without this “backward compatibility”, similarity searches will return subpar answers.

Embed 3 processes both text and image data in a balanced way, preserving their semantic meaning in a shared space. Unlike other models that tend to separate text and image data into distinct clusters (leading to results that favor text over images or vice versa), Embed 3 maintains equal performance across both types of data. This means when you search across mixed content types, you'll get the most relevant results regardless of whether they're text or images.

Before committing extensive resources to multimodal embeddings, it’s a good idea to test it on a more limited scale. This enables you to assess the model’s performance and suitability for specific use cases and should provide insights into any adjustments needed before full deployment.

Models may need additional training to pick up fine-grain details and variations in images. A more specialized multimodal embedding system may be needed for medical applications, for example, where it needs to understand images like radiology scans or microscopic cells.

The output part of the pipeline requires additional considerations. The system should be able to process image pointers (e.g., URLs or file paths) alongside text data, which may not be possible with text-based embeddings. To create a smooth user experience, organizations may need to implement custom code to integrate image retrieval with existing text retrieval.

Obtaining seamless generation results that integrate image data alongside text will also require updated reranking models. Text-based reranking models will overlook relevant information from images (aside from metadata) because the meaning is tied up in the image itself and because conventional rerankers accept text, not vectors, as input.

What steps should be taken to prepare data for multimodal embeddings?

As with text-only embeddings, multimodal embeddings require data to be pre-processed to some extent before being fed into the model.

Images, for example, will usually need to be resized to a consistent size and scale. Attention also needs to be paid to the quality of the images. Low-resolution images could lead to a loss of important details. On the other hand, high-resolution images (which might be necessary for some use cases, such as medical ones) will put more strain on processing time and memory. Organizations will need to strike a balance based on their needs.

Ensuring that detailed metadata is associated with images is another important step to improve the relevance and context of retrieval results.

What's the process to evaluate performance?

A combination of human and automated evaluation techniques can be deployed to evaluate and adjust multimodal performance.

Workers can compare the embedding results with a traditional keyword-based search for images. Organizations can also perform A/B testing by deploying two embedding versions and comparing their performance on retrieval and other metrics.

An important part of performance is understanding how well you can process the embeddings and scale the system as an enterprise-grade model. This can be a challenge, particularly for businesses like retailers that may need to embed a massive volume of items. It can be done, though. A major retailer that partnered with Cohere was able to index over 700 million items in four days, transforming its customers’ search capabilities. However, processing times can vary based on factors such as infrastructure capacity.

Conclusion

Multimodal embedding is poised to be a major leap forward for enterprise-grade GenAI applications, enabling powerful AI search across a full range of data and generating more insightful, comprehensive, and visually rich responses to queries. As different industries refine their use cases, we’re likely to see more integration of formats such as video and audio and improved multimodal performance for highly specialized use cases in sectors such as retail, healthcare, and manufacturing.

Building enterprise AI agents

A comprehensive blueprint for developing scalable AI agents in regulated industries.

Download the eBook

Multimodal embeddings: Unifying visual and text data