With the rise of supermassive context windows (1+ million tokens), there's an evolving narrative about it being a "RAG Killer." Long context is great as it enables more use cases, but that doesn’t translate to the end of retrieval-augmented generation (RAG). In fact, it’s not an either/or decision for enterprise builders.

Larger context windows make everything better, including RAG. The downside is the added computational load of processing massive context windows, which leads to increasing costs and latency.

But, if we take a step back and just look at the architectures for processing information with either massive context or RAG systems, we see that their underlying approach is very similar in practice. Essentially, they both require focusing attention selectively to manage and interpret vast amounts of data efficiently. The difference is that with RAG come several added benefits optimized for enterprise use cases.

This article highlights four reasons why RAG is here to stay, and why it is an optimal solution for enterprise use cases, with and without massive context windows.

Large context windows still don’t scale enough: Large context windows often struggle to scale effectively to meet extensive data needs, generating more costs and latency. This makes RAG a more viable option for handling large-scale information processing.
RAG outperforms in enterprise use cases: Optimized RAG systems are often more effective and deliver better quality for a broader range of data sources, even with caching mechanisms on context windows.
RAG delivers a reliable source of truth: RAG ensures that the data used is accurate and up-to-date, acting as a source of truth, which is essential for building reliable solutions.
RAG leads to agentic capabilities: RAG naturally extends to the use of more complex executions across multiple data sources, enabling AI agents to pull in diverse insights and provide more comprehensive responses and solutions.

Before we dive in, it’s important to point out that for both large context windows and RAG systems, you’ll want to use a model that has been heavily trained to handle and interpret results from data retrieval. If not, when exposed to an excessive amount of documents, a model may saturate, meaning its performance plateaus or declines, and it may even start to produce confusing or irrelevant outputs. Proper training is crucial to ensure that the model can manage and benefit from large volumes of data.

Large context windows still don’t scale enough

While larger context windows allow LLMs to process more information, this comes at a significant computational and financial cost. Processing long context lengths is super expensive (even for the compute-rich players like Google), and super slow — which can be tolerated in some, but not many cases. For example, a context window of 10 million in length can take almost a minute to generate a single response given a few million tokens.

RAG, on the other hand, is more efficient as it only retrieves the most relevant information for a given query, reducing the number of tokens that need to be processed. This makes RAG a more cost-effective solution, especially for applications that require frequent or high-volume queries and information intensive tasks.

RAG outperforms in real enterprise use cases

Serving models with longer context lengths is a very difficult task, and it is far from perfect. In fact, there are examples where models can struggle to leverage their massive context and even show significantly degraded performance. Recent research using needle-in-a-haystack benchmarks revealed effective lengths to be much lower than previously thought.

There are on-going explorations to carry out long-context without decreasing performance using methods like caching. In the context of LLMs, caching refers to the process of storing and reusing previously generated responses or retrieved context from an LLM to improve efficiency and reduce latency. Simply put, it reduces the computational load. This is most apparent in use cases with many repetitive and overlapping queries.

There is a case to be made for larger context windows with caching. Once caching is implemented and the model can fit the entire needed context (for example, a book or catalog of books), large context windows would be much simpler to use compared with RAG. For example, in summarization use cases, a longer context window with cache may provide the best solution for some customers.

The challenge comes with scale and usability, when caching longer and more dynamic data, context windows may not be a viable option. For example, if you want to cache one million 1-2k dimensional vectors, that's an enormous amount of data. This means that you'll need to keep this data in "hot storage," which is a type of storage that allows for quick access, particularly important for tasks that require rapid data access, such as real-time data processing or high-frequency trading. This will quickly add up to higher costs and more latency.

While caching may help mitigate some of the performance issues with long contexts, an optimized retrieval system offers a smarter, more efficient, and cost-effective way to manage and access a large variety of datasets. RAG not only improves performance, but it also supports broader strategic objectives like the scalability and adaptability needed for real-life enterprise scenarios.

RAG delivers a reliable source of truth

Grounded generation (the part where we process chunks of text and generate a response with citations included) becomes more important, not less, with longer context windows. RAG's retrieval process provides transparency, revealing the reasoning behind the generated responses. With RAG, the source of the information can be traced back to the retrieved documents and citations can be provided.

Source attribution can be important for applications that require accountability or interpretability, such as in regulated industries or high-stakes decision-making. With only a large context window, it's unclear what information the model is basing its response on from the provided context, making it difficult to audit.

RAG systems can also provide a layer of explainability and control over the generated output by controlling the information sources used for retrieval, ensuring that the model only accesses trusted and relevant data. For most enterprise use, including domains like finance, healthcare, and law where accountability is paramount, this becomes ever more relevant to verify for accuracy, completeness, and relevance. In contrast, large context windows can provide a wealth of information, but may also include irrelevant or tangential details that can negatively impact the quality and accuracy of the response.

In simple terms, if you're going to process 10 million tokens, you will need citations, sometimes called groundings. Otherwise you have no way of knowing how, or even if, the model leveraged the information in your massive context.

RAG leads to agentic capabilities

RAG opens the door to more advanced cases of retrieval from multiple sources, allowing different users access to different sources and actions. For example, building a RAG application with tool use that can incorporate personal preferences or account information — a particularly useful feature for e-commerce, recommendation systems, and personalized assistants. This flexibility is difficult to achieve with a one-size-fits-all context window, which may not be able to capture the nuances and complexities of diverse enterprise use cases.

On the flip side, large context windows are great at digesting heaps of information all at once, which comes in handy for complex thinking tasks. But they are limited to the generic information present in the model's training data, meaning that when it comes to creating truly personalized experiences based on very specific needs and preferences, they might not hit the mark.

That’s where RAG systems shine — they’re super flexible and adaptable, making them ideal for a variety of complex and dynamic tasks. And personalization is just one example. There are endless possibilities for AI agents and agentic behaviors that RAG with tool use allows.

Large context windows are pretty amazing when it comes to understanding complex information, but for most enterprise use cases, they’re not enough. RAG quickly becomes a must-have, especially when it comes to efficiency, performance, accuracy, and flexibility. In the future, what approach you take will likely be determined by how large the grounding dataset is. This means that the volume of data you can access and use to train or inform your systems will play a crucial role in determining which methods or techniques are most appropriate. As AI evolves, understanding both RAG and large context windows is increasingly important, helping us tackle the wide array of needs from different applications and scenarios.

Ready to start building? Here’s a cookbook and walkthrough video.

Let's talk

RAG is here to stay: Four reasons why large context windows can't replace it