Generative AI has entered the public sector around the world not with a bang, but with a cautious knock at the door. Across government departments and agencies, there is growing recognition that AI could — indeed, must — play a key role in modernizing service delivery, improving policy development, and increasing efficiency. But real deployment is still the exception, not the norm.

The potential is enormous. In the near future, McKinsey projects GenAI will unlock $100 billion in productivity value in the public and social sector. The UK Government also estimates that applying AI to routine tasks across the public sector will be “worth billions”. Still, the necessary momentum is yet to build.

In many agencies, AI remains stuck in pilot mode — limited proofs of concept that struggle to move forward. These efforts are often blocked not by lack of ambition, but by a familiar set of institutional hurdles. What’s emerging now is a clearer picture of what’s actually slowing things down — and what it will take to move past them.

What’s slowing things down?

Across governments, the same issues keep cropping up — structural, familiar, and frustrating. They’re not about whether AI is important, but about the barriers that prevent it from working at scale.

1. Procurement inertia

Procurement is structurally tilted toward the status quo. Legacy vendors benefit from existing contracts, name recognition, and well-worn pathways through government red tape. Newer providers, no matter how capable, are often held back by risk aversion and rigid processes that leave little room for experimental partnerships. Similarly, lengthy certification and compliance timelines often mean the most innovative providers are left waiting years even to attempt to work with the public sector. The result is a market dominated by a few familiar names. According to the 2023 Outsourced and Automated report from the Electronic Privacy Information Center, just 10 vendors account for over $715 million in AI contracts across 42 U.S. states — a clear sign of how procurement systems, designed for stability, shut out agility. Empowering agencies to use more flexible, agile procurement mechanisms tailored to fast-evolving tech will be critical in this next phase of adoption.

2. Talent starvation

Public agencies are in a chronic state of talent loss. Even when they manage to recruit skilled people, the private sector swoops in offering irresistibly higher salaries. This is doubly true in the AI sector, where there is a scarcity of talent around the world. In 2024, the UK’s National Audit Office report found that the lack of skills was a barrier to AI adoption in 70% of government bodies. This brain drain leaves teams unable to build or maintain in-house capability, pushing them toward off-the-shelf AI tools that may be easier to obtain, but are far harder to adapt to the required use cases.

3. Dead-end pilots

Governments have run hundreds of AI pilots. Most don’t fail — they just fade out. There are a number of reasons for this — from a lack of definition on use cases to clear success metrics on the problems the technology is trying to solve. Without champions to push for adoption, these projects drift into limbo. According to a global Capgemini report, just 9% of public sector organizations had scaled AI beyond the pilot stage or small-scale deployment — well behind sectors such as life sciences (27%) and retail (21%).

4. Burned bridges

When AI gets it wrong — mangles an acronym, misinterprets a form, hallucinates an incorrect answer — it chips away at trust. Even minor missteps can trigger disengagement, especially in high-stakes environments like healthcare, welfare, or legal aid. Users switch off. A feeling of “AI doesn’t work here” sets in. In the public sector, AI tools must prove their value immediately. Accuracy and trust are critical.

5. The weight of cost

Even free open-source models aren’t truly free. Running them demands infrastructure — GPUs, memory, energy — that many agencies simply don’t have. For adoption to scale, governments need tools that are not just accurate, but affordable and efficient enough to deploy across departments and time zones.

What works: Small wins, real ROI

Rather than aiming for sweeping transformation, public agencies are beginning to succeed with focused, practical use cases — the kind that solve immediate problems and demonstrate value quickly. These include redacting sensitive documents, summarizing complex reports, aggregating policy inputs across departments, and improving citizen self-service for routine queries.

Much of this early progress comes from targeted, retrieval-augmented generation (RAG). RAG allows a model to access current, secure government data at query time without ingesting it — meaning outputs are more accurate, more current, and less prone to hallucination. It also allows for controls: user permissions, in-line source citations, and audit trails that align well with public sector imperatives.

The emphasis is on supporting public servants and improving their efficiency, not replacing them. And as these use cases demonstrate their value, they begin to build something more foundational: trust.

Build trust with transparency and security

Governments operate under intense scrutiny and are often responsible for highly sensitive data, so their decisions must be explainable, and their processes demonstrably fair.

Public confidence in AI remains fragile. Ipsos research in 2024 found that while awareness of AI is high in the UK, trust and understanding remain low. To build that trust, public bodies are placing greater emphasis on verifiability, transparency, and operational control. For example, in 2024 the U.S. Bipartisan House Task Force on AI highlighted at the top of its list that responsible government use of AI focuses on accuracy, reliability, and robustness.

To meet these expectations, agencies need systems that can explain their outputs, cite their sources, and demonstrate a clear chain of reasoning. At the same time, public trust hinges on how personal data is handled. Privacy is paramount. Governments must ensure that AI systems collect and use only the data that’s necessary, comply with requirements such as HIPAA and FedRAMP in the US, and avoid exposing sensitive information through training processes or user queries.

This means private deployment — running AI models entirely within a government agency’s secure environment. It keeps data local and under agency control, significantly reducing privacy risks. When combined with RAG architectures, private deployment enables governments to meet high standards for data protection, auditability, and data sovereignty.

Efficiency: The silent enabler

As AI models improve, size is no longer the sole marker of capability. Efficiency — the ability to run models quickly, securely, and affordably — is increasingly critical.

This is especially true in the public sector, where hardware constraints and tight budgets are a constant reality. Until recently, deploying a high-performing model often meant investing in infrastructure capable of supporting 16 to 32 GPUs — an expensive, energy-intensive undertaking. Newer models, such as Cohere Command A, change that equation. Designed for enterprise and government use, it delivers performance on par with, or better than, systems like GPT-4o — while running on just two GPUs.

That stunning level of efficiency not only lowers costs, but also enables AI deployment in places where traditional models can’t go. For defense and national security applications, that means on-premises systems and edge environments in the field — not just in big data centers. In field settings, power, latency, and reliability become a mission-critical advantage.

Just as significant is the growing importance of fixed, predictable pricing. Many agencies have been caught out by usage-based models that scale unpredictably as adoption grows. By contrast, stable pricing allows government agencies to budget confidently, test new applications, and expand successful ones without fearing spiraling costs.

For agencies under pressure to deliver both innovation and value, these shifts open new doors. AI no longer needs to be prohibitively expensive — or restricted to flagship deployments. Efficiency unlocks scale.

Execution and partnership: What comes next

There’s no shortage of interest in AI across the public sector, but interest is not adoption. To move from pilots to production, agencies need more than one-off product purchases. They need trusted tools, clear outcomes, efficient infrastructure, and the right partners.

Public–private collaboration is central to this next phase. Agencies may need help identifying optimal use cases, integrating with legacy systems, ensuring compliance, managing security, and evaluating impact. Increasingly, the value lies in collaboration, not just code.

A collaborative development model between government and industry can allow agencies to leverage commercial AI expertise while maintaining appropriate security control. Promoting diverse pilots for a given use-case (so multiple solutions are evaluated), and then adopting the best and most flexible solution is essential.

Governments are looking for partners who can work across the lifecycle: from discovery and design through to deployment and scale. Those partnerships must be flexible, transparent, and focused on ROI.

The most effective applications may not make for dazzling headlines, but they can reduce wait times, streamline back-office workflows, and help overstretched teams surface the right information, faster. These aren’t AI moonshots. They’re foundational improvements. And they’re within reach — if governments can execute with focus and confidence.

Mastering private AI deployments

Is your business ready for a secure, private AI deployment?

Download the eBook

Sohail Manoussi