As enterprises tap into their vast knowledge bases to leverage generative AI, mastering the critical skill of chunking has become essential for optimizing retrieval-augmented generation (RAG) systems.

Below we answer some of the most common questions our team gets from customers about chunking. Let’s dig in.

What is chunking and why is it so important?

Chunking is the process of breaking down large pieces of information into more manageable and context-rich chunks. This makes it easier for AI systems to process, search, and retrieve the most relevant information.

Essentially, chunking organizes information into smaller, more semantically coherent units, improving the AI model’s ability to locate and retrieve the most relevant answers. Chunking typically takes place during pre-processing, where large documents are broken down into smaller sections, such as paragraphs or sentences.

Chunking works by improving the quality of embeddings, the numerical representations of data that a GenAI model can understand and use. When each chunk focuses on a specific piece of information, it makes it easier for the embedding model to create more accurate and detailed numerical representations of the data stored in vector databases. High-quality embeddings can make the difference between a RAG system that delivers on or exceeds expectations and one that falls short.

When should you consider chunking?

Chunking is a valuable step for any organization that needs to extract and process information from large documents and data stores as part of a GenAI solution. Breaking content into smaller, more manageable sections improves both the relevance and accuracy of retrieval results.

Because chunking results in fewer parts of documents being sent to the LLM, it also reduces costs and latency. The success of chunking within RAG systems lies in finding the right balance between chunk size and retrieval precision.

What size context window should I use?

The context window is the maximum amount of text a model can process at a given time. Context windows are measured by tokens, the building blocks of text such as a word or symbol, and most large language models feature an upper limit.

Chunks are created to fit within the context window of a model. Setting the right context window for a particular use case is crucial to maximize the effectiveness of RAG systems, playing a key role in determining the accuracy and relevance of results, and a model’s overall performance. Set it too small and the chunks won’t have enough semantic connections with the rest of the document for the model to understand its meaning. Too big and the model won’t have enough specificity to retrieve the most relevant and accurate information with which to generate responses.

Limiting the size of the context window and then adjusting it based on retrieval performance is the recommended approach. Cohere Embed, for example, has a maximum chunk size of 512 tokens. If the documents you are embedding have paragraph lengths of around 200 words, for example, a chunk size of 250 words might be appropriate. This overlap, or “sliding window” approach, helps ensure you are collecting some context from the previous and subsequent text. In general, a smaller context window (100-300 words) is better in order for the model to more accurately capture the semantic meaning of text. Cohere’s research and experiments have found that performance is broadly optimized with a few hundred tokens and drops significantly after around 500.

What are the different text chunking methods?

For most companies, a chunking strategy will need to include a range of different data types. For unstructured text, here are some common methods we’ve seen to be robust for enterprise customers:

Fixed Size. Content is divided into predetermined chunk sizes. For example you might choose to chunk all documents into 512 token segments. This approach is simple and cost-effective, but it can lose context and is not suitable for smaller documents.

Sentence Level. Text is broken down into individual sentences, allowing for a more granular analysis. This provides a detailed understanding of each sentence's semantics without context from the rest of the text. For this reason, it is ideal for shorter queries or when the focus is on specific sentences rather than broader themes.

Paragraph Level. This method retains more contextual information than sentence-level chunking. It is more suited to longer queries that span more than one sentence or a paragraph, as it is likely looking for broader semantic understanding. This method doesn't require overlap with previous chunks for semantics.

Sliding Window. This technique uses a 'window' of text that slides across an entire document, capturing and processing a set number of consecutive sentences. Unlike paragraph-level chunking, this often creates an overlap with previous chunks to ensure semantic coherence. This is an effective method for processing longer queries as it maintains a broader context to capture the semantic meaning and relationships within the document.

Recursive Chunking. Text is split up into smaller pieces at natural breaks, such as the end of sentences or paragraphs, rather than cutting it at fixed intervals. This makes the text easier to read and understand, and preserves the meaning. But it is more complex to set up and can be tricky to manage due to the creation of chunks of different sizes.

How do you chunk structured documents like tables?

Structured document formats, such as tables in PDF files, can pose a different chunking challenge. Each value in a table row or column may be meaningless without its associated headers or neighboring entries. Failing to chunk a large table correctly will lead to a loss of semantic connection and ultimately retrieval results that are out of context or misleading.

If the chunk only consists of the middle portion of the table, for example, the model will struggle to understand the relationship of column entries to the headers. A specialized chunking strategy can be deployed to segment the table in a way that preserves headers with each chunk so that the entire table can be reconstructed during retrieval. It’s vital to include as much metadata (such as table headings, keywords, and dates) as possible in the chunks. This enables the RAG system to understand the context of each chunk more accurately and to narrow them down based on the relevance of their metadata to the query.

What tools can I use for chunking?

Chunking large troves of company data can be a hefty, time-consuming, complex task. So it helps that there are tools available to make it faster and easier to both prepare documents for chunking and also implement the chunking itself. Tools like Unstructured, LangChain, and LlamaIndex, for example, enable you to extract data based on its structure type and chunk it appropriately. Take the example of a research report stored as a PDF file containing large amounts of text but also a number of tables. The tool can extract the data from the PDF and assign different labels for text and tables to allow for more targeted, efficient chunking.

The initial processing of large-scale data can take anywhere from several hours to days depending on compute resources. Subsequent updates can be processed much faster, however, with these tools able to extract and chunk new documents in a matter of seconds or minutes. The tools are designed to scale horizontally, allowing for distributed processing to handle larger workloads.

How do I refine my chunking strategy?

Chunking requires a continuous process of testing and iterating different strategies to see which one meets your unique needs based on use cases and data types. In order to optimize chunking, it’s necessary to evaluate how chunking is affecting search performance, and to adjust and refine your strategy accordingly.

As a starting point, the model’s performance can be assessed manually based on the relevance of the chunks being retrieved in response to specific queries. But it’s important to implement more detailed metrics for a comprehensive evaluation. Recall@k and Precision@k are the most common metrics used to measure chunking performance. The formula used for Recall@k measures how many relevant chunks were retrieved, while Precision@k measures how many of the retrieved chunks are relevant. An F-1 score can then be calculated that balances both precision and recall performance.

Another key metric is Mean Average Precision (MAP), which gives an overall view of ranking quality across multiple queries. Normalized Discounted Cumulative Gain (NDCG) is particularly useful for evaluating the quality of ranked retrieval results, taking into account the position of relevant chunks. By systematically tracking these metrics, you can gain deeper insights into your chunking strategy's effectiveness and identify areas for refinement. This is especially useful when scaling to large production where performance characteristics may shift.

Based on the metrics, you can tweak your approach by, for example, increasing or decreasing the chunk size or the sliding window of each chunk and measuring the impact on retrieval results. You may need to include more, or different types of, metadata.

Getting chunking right is proving to be a challenge for many organizations. Chunking perfection will rarely, if ever, be achieved at the first attempt. The optimal strategy will depend on use cases, model capabilities and the type of data being used, and needs to strike a balance between preserving the context of documents with optimizing retrieval efficiency. Breaking down vast stores of internal data held in dozens of different file types into bite-sized chunks can seem like a daunting task. But with the right tools, know-how, and partners, it doesn’t have to be.

Are you ready to start building with AI? Get in touch with our experts today.

Get Started!

Kasim Patel