Ada, the trusted leader in agentic customer experience (ACX), helps other businesses — from airlines to payments to retail — autonomously resolve support customer service inquiries. Behind these actions are AI agents that each customer configures with its own knowledge, brand voice, and processes, running on Ada's Reasoning Engine. For instance, the type of answer a financial services company gives a banking customer will be different than the one an airline gives a traveler whose flight was just cancelled.
But before AI agents can generate the right answers, they must find the right information. That’s where retrieval comes in. Every customer interaction begins with retrieval, giving AI agents the information they need to resolve conversations.
“If retrieval is slow, our agent is slow and everything that stems from it... and if retrieval is wrong, then the agent is wrong as well,” explains Christos Melidis, PhD, Staff Machine Learning Scientist at Ada.
The initial challenge
Ada’s online use case: every millisecond counts
Now imagine you are calling in, asking an AI agent a question and hearing an unnatural pause before every answer. A few milliseconds might sound insignificant on paper, but inside a conversation, you feel it. A fast retrieval model gives customers more confidence that their questions will be answered in a timely manner instead of wondering whether the call dropped.
“If you're on the phone, you don't have a visual. You don't see an indicator blinking. You're hearing silence,” explains “The smaller the silence, the better it is.” In other words, when voice enters the chat (literally), the stakes are higher.
Three requirements
Within the online use case, Ada set a tighter latency budget for voice than its previous retrieval setup could deliver consistently. The solution had to meet three requirements:
Quality: Find the right information in each customer's knowledge base
Latency: Fit within an end-to-end voice latency budget that left little room for retrieval
Scale: Perform across regions at multi-tenant scale
A structured approach to find a solution
Ada conducted a structured evaluation to find a solution for both offline and online use cases. "We started in a very structured way—a four-way bakeoff,” explains Melidis. The bakeoff, a systematic, time-bound testing period, vetted each approach against both of Ada's workloads.
The bakeoff’s ingredients:
15,000 evaluation queries
Online and offline workloads
Four embedding approaches
Three requirements: quality, latency, and scale
One north star: successful resolution
From the start, the testing challenged one of Ada’s assumptions: that a single deployment could efficiently serve both workloads. Two use cases demanded two technology solutions.
Testing reveals a key insight: two use cases, two solutions
For offline knowledge-base indexing, Ada uses Cohere Model Vault. Dedicated, Cohere-managed infrastructure runs models privately and at scale, and supports bulk embedding and ingestion. For its online retrieval path, Ada self-hosts Cohere Embed
on dedicated Amazon SageMaker AI instances. Cohere Embed converts content and queries into numerical representations that make it possible to search for semantically similar information.
“We saw better performance, better quality in the answers, and a better experience for the clients.”
Christos Melidis, PhD, Staff Machine Learning Scientist, Ada
The recipe for success emerges
In the online use case, Cohere Embed quickly emerged as a star, achieving 90.7% top-10 recall. Because SageMaker AI was geographically closer to Ada’s services, it was the sensible choice for the latency-sensitive live conversations. With Cohere Embed on SageMaker AI, Ada achieved median retrieval latency of 77 milliseconds (ms) — only 0.077 seconds for that step in the response process. For a caller, shaving time from retrieval helps reduce the silence between asking a question and hearing the agent respond.
For offline workloads, Model Vault delivered approximately 2.7x greater throughput-per-dollar than Ada's self-hosted setup. Its managed autoscaler handled republishing spikes faster than Ada's self-managed scaling.
Across the retrieval system, Embed enables compression without performance impact. Because Embed supports compression natively, Ada did not need to build its own quantization pipeline. At Ada's scale — thousands of customer knowledge bases, some containing tens of thousands of articles — results quickly compounded. The team reduced its vector storage footprint by 4x with no measurable retrieval degradation. "It was literally a free win,” says Melidis.
Better retrieval gave Ada's agents better information to work with, supporting its north star metric: successful resolution.
A partnership rooted in customer success, now and for the future
Through the process, Cohere provided hands-on benchmarking and experiments to find the right configuration, helping solidify Cohere as a trusted retrieval partner for Ada. Says Melidis, “Whether we were using SageMaker or Vault, we felt like there wasn't a strong pressure. The teams just wanted us to be successful.”
The Ada team has also seen clear benefits from running its embedding models on dedicated deployments rather than multi-tenant services, particularly for latency-sensitive workloads. Says Melidis: “Dedicated compute for embedding models was new for us. This change showed how much a dedicated deployment matters for retrieval when latency is of utmost concern.”
Today, Ada can use its existing AWS environment for deployment flexibility, performance, and operational ease. Cohere Embed could fit into its existing architecture, allowing the team to optimize retrieval while preserving the models and infrastructure already supporting its agents.
The project gave Ada more than a point solution. It established a retrieval architecture that the team can continue to evolve as its AI stack changes. With Cohere already preparing new search models, optimizations, modalities, and regional deployments, Ada is ready to keep testing what comes next.
