Updated: June 3, 2026

Generative AI (GenAI) systems have the potential to transform company operations, but they’re only as good as the data they can access and use.

Therein lies one of the biggest challenges for enterprises as they seek to harness the power of AI: How to effectively connect to and retrieve data from sources that often vary extensively in quality, format, size, structure, and even language.

To be sure, connecting to reliable, consistent, timely data has long been crucial to getting the most out of enterprise software applications. But to reap the benefits of advanced enterprise AI systems like retrieval-augmented generation (RAG), companies need to raise the bar. The adage “garbage in, garbage out” becomes truer than ever.

The exact approach to tackling data challenges will vary widely depending on a company’s industry and size, and their overall goals for AI adoption. The first step, therefore, is always to establish a clear idea of the use cases for RAG systems.

Some companies, such as those in financial or professional services, might predominantly need access to structured data such as client-specific information from CRM systems, financial and market records, and regulatory compliance data. Others might need to retrieve more unstructured data, such as emails and social media posts, for which language generation is more important than analysis.

With a clear understanding of the data requirements, enterprises can start to tackle some common data challenges. Here, we identify the top three challenges and how solutions like embeddings, data connectors, and reranking can help ease many problems. Let’s dive in.

Addressing poor data quality

Poor data quality has become the primary factor limiting AI adoption, according to consulting firm Forrester. Machine learning models require good data quality for effective training. Generative models have similar requirements – not only during their training, but also when they interpret and synthesize data to generate responses.

In most applications of RAG, a search system retrieves relevant data (textual, tabular, etc.) based on a user query. The quality of the data available directly impacts its searchability and also how well the generative model is able to interpret and build responses based on it. Noisy data – for example, content sources that contain multiple mistakes or gaps – is more likely to lead to inaccurate or irrelevant results, reducing the effectiveness of a system and undermining trust in it.

The issue of data quality can be tackled at different levels. Before connecting data with an AI engine, companies can undertake preprocessing, or cleaning, to address any gaps, errors and inconsistencies in their documentation. Cleaning can also remove sensitive data, such as personally identifiable information (PII), that could create legal or reputational risks if fed into the AI engine.

Choosing a GenAI provider with a robust search solution is a crucial part of quieting data noise. The most effective search systems use hybrid search, which combines an embedding model for semantic, context-driven retrieval combined with lexical matching for precise, keyword-based queries.

Strong embedding models are compatible with a variety of data formats and structures, such as structured (tabular), semi-structured (JSON, XML etc.) and unstructured (text). The best embedding models are trained not only on semantic information, they also reserve numbers in each vector for content quality, structure, and size. This sharpens AI model responses by focusing on higher quality and more relevant data. For additional precision, semantic rerankers can be deployed to order retrieved data based on their relevance to a search query.

What about when documentation contains conflicting information? It may not be possible, or desirable, to expunge all conflicting data. One option is to allow the large language model (LLM) to sift through the data and generate an answer. Another approach would be to tailor the LLM to inform users when a conflict arises and provide different potential answers. Models that include citations enable users to immediately verify the documentation being used to generate an answer. Human experts can guide the LLM to value one type of data source over another, or data with more recent time stamps or written in a certain tone (more neutral, for example) to improve the accuracy of results.

Managing data held in different formats and locations

Data noise also gets amplified by the wide range of structures used to store documentation, from SQL tables to Excel spreadsheets, to CSV files, to XML files. In addition, a lot of data tends to be stored in unstructured formats like emails, PDF documents, and PowerPoint files. If an AI system isn’t able to cope with a wide range of data formats and locations, its effectiveness and capacity to scale will suffer.

Careful consideration needs to be taken for how to access and index various data formats. Once you know what data you need to access, it needs to be made available to the AI model. This requires setting up data connectors that connect an AI model to the relevant document sources and structure type.

The most common enterprise datastores, such as Google Drive, come with pre-built APIs that businesses can use. For less common sources of data, enterprises may need to build their own custom APIs or use an AI provider that includes data connectors as part of their service. Cohere, for example, has built about 100 quickstart APIs that can be used to connect the most popular datastores to our frontier enterprise-grade models. In addition, Cohere provides a template that organizations can use to create custom data connectors.

When documents are well organized, they are easier for the generative model to understand. Documents stored in frameworks like Confluence or Jira, for example, require certain common structural elements to improve their quality and readiness for AI processing.

Enriching documents using metadata is another way to bring more order and value to data held in varied structures. Many document types will already include metadata – such as the author, creation, date, and update date – that allow for more precise searching. A strong reranking solution can prioritize metadata fields and provide a powerful additional layer of accuracy and relevance. Cohere Rerank, for example, natively supports metadata-based searching. Structuring documents in the JSON (JavaScript Object Notation) file format can be a good solution for leveraging metadata because its flexibility allows it to store diverse data types. The fact that JSON files are human-readable also makes it easier to check and debug documents.

A strong reranking solution can also rank complex, multi-aspect data, such as emails that include a combination of metadata, attachments, and body text. Adding Rerank to AI search systems has demonstrated significant data search accuracy and performance improvements compared to systems without a reranker. This includes better retrieval of code, enhancing organizations' ability to extract relevant information from large code repositories. Rerank passes only the most relevant documents into your RAG pipeline and agentic workflows — reducing token use, minimizing latency, and boosting accuracy.

Balancing data cost at scale

As organizations seek to expand their AI operations, data bottlenecks often emerge as a hindrance, affecting the speed, efficiency, and cost of the system. The noisy data challenges mentioned above are part of this – scaling magnifies problems in data quality and structure. The sheer volume of data needed to run large-scale AI systems can also create headaches, demanding more computational and storage capacity, which can quickly ramp up costs.

To limit costs, improve accuracy, and reduce the overall workload of your AI solution, organizations need a strong retrieval system optimized for scale.

Start by using embeddings to process large, complex datasets (several hundred gigabytes, for example) more easily. Embeddings convert high-dimensional data such as text and multimedia files into dense vectors. Consider using compressed embeddings, which drastically reduce the storage space required for vector locations by storing them in fewer digits.

There can be a trade-off here, though. Compressing embeddings can degrade the quality of the search because it reduces the enrichment of the vectors. It’s therefore important to select an embedding model that creates a minimal amount of degradation.

For more precision and relevance, and to significantly reduce the workload and costs of AI solutions, use reranking tools, like those mentioned above, with metadata-based search. It reduces the amount of data that requires expensive processing, optimizes when and how AI models are invoked, and lowers both computational and storage costs.

And lastly, implementing an effective chunking strategy is crucial for indexing various data formats. Breaking down diverse documents and datasets into manageable, uniform chunks enables RAG systems to retrieve information more efficiently and enhances response accuracy.

Conclusion

Bringing order to today’s data chaos is a foundational challenge for organizations seeking to unlock the potential of advanced AI systems. Each company will have unique data challenges. But there are some common ingredients for data success, including the smart use of data cleaning, embeddings, data connectors, and reranking to ensure the AI model retrieves and generates answers from high quality sources. Importantly, data discovery shouldn’t be treated as a “one-and-done.” It requires continuous refinement of both data handling and the tech infrastructure around it to ensure high quality results that meet an organization’s strategic goals.

Mastering private AI deployments

Is your business ready for a secure, private AI deployment?

Download the eBook

Sunith Suresh