For many enterprises across industries, the use of artificial intelligence has moved from a curious experiment to a necessary expense, with their digital transformation strategies growing alongside AI innovation. Yet, many companies still do not fully understand the unit economics of what they are getting when purchasing AI technologies, which can turn into a financial risk as AI workflows scale in ambition and become a permanent part of conducting business.
At Cohere, we recognize our customers’ challenges related to the total cost of AI ownership (AI TCO) and the strategic decisions over when to own AI technology versus when to rent it — and AI ownership is not one size fits all. In this post, we’ll walk through both the invoiced and hidden costs of running AI in your enterprise and how to approach your AI ownership strategy.
To connect with Cohere sales to see your total cost of AI ownership, and discuss opportunities to lower it, please contact us.
How AI costs are calculated
The visible price of AI is usually quoted in tokens. Tokens are the fragments of text that a model both reads and generates. Tokens are incurred across every prompt, response, retrieved document, agent step, tool call, loop, and retry. Yet token pricing is only one part of cost — albeit the most visible and unpredictable given the way that AI usage scales when effectively implemented within an enterprise.
The larger question, however, becomes not what a vendor charges per million tokens, but what it costs to own, operate, secure, and scale AI once it becomes an indivisible aspect of how business is conducted. This is the core challenge that AI ownership presents to organizations.
AI TCO is rising alongside usage
Today, the total cost of ownership is increasing as enterprise AI becomes a major business expense. Gartner projects global AI spending will reach $2.52 trillion USD in 2026, a 44% annual increase, driven primarily by infrastructure. It matters more as AI adoption increases, because AI maturity and usage results in more inference, longer context windows, more agentic steps, increased retrieval, and demand for speed.
One answer on a screen may represent a chain of model calls, document searches, tool invocations, and validation steps — and these steps are not hidden in infrastructure or model invoices.
Layers of AI ownership: Renting and owning infrastructure
As AI becomes a standard business cost, firms must understand what they are purchasing: tokens, latency, throughput, utilization, data control, and the ability to forecast expenses. Enterprises, specifically financial leaders and those close to AI transformation initiatives, must focus on which variables drive costs and which ensure control.
Firms that scale AI profitably will not be those that spend the most, but those that work out which AI capabilities to own and which to rent — and build the discipline to tell them apart. Those that rent everything reach the same conclusion eventually, when the cost grows to become unsustainable or the control quietly slips away.
The hidden costs of AI
Token economics is the obvious starting point because it explains why AI spend behaves differently from that of traditional software. In conventional cloud computing, costs map to familiar units: servers, storage, bandwidth, and time. AI introduces a more volatile unit of account.
A token is small, but the systems built on top of tokens are not. A single user request can trigger a long prompt, large context window, multiple reasoning steps, several tool calls, and a polished final answer. The user experiences one interaction; the company pays for the entire chain.
This is why AI costs can surge without visible product changes. Teams may expand context windows for better answers, add retrieval to reduce hallucinations, introduce agent loops for complex tasks, or route requests to larger models — all sensible decisions that, in combination, dramatically alter the cost profiles.
Looking at the AI system at large
The common mistake is treating token price as unit economics. It's not. Total cost of ownership is the system that produces the token. The relevant questions are broader:
- How many tokens does a task consume?
- How many model calls sit behind a user action?
- Which model handles each step and how is it routed?
- How much context is passed?
- How often does the system retry?
- What latency does the business require?
- Is the paid-for infrastructure actually being utilized?
A great deal of this is driven by infrastructure, but also by the imperative that companies must be on the cutting edge of digital innovation in order to stay competitive. As such, companies tend towards the consumption of the largest models that are at the forefront of commercially available innovation. For example, an earnings call from Uber stated that “10% of the company’s committed code is built by autonomous agents,” which coincided with the company spending 12 months of its AI budget in only four.
Headlines abounded, but the question remained: AI created more productive organizations, but for leaders concerned with revenue forecasts, innovation cannot come at the price of inaccurate forecasting.
The surprise is already visible across markets
Recent research highlights the scale of unexpected AI infrastructure costs. According to a December 2025 IDC InfoBrief commissioned by DataRobot, 96% of organizations deploying generative AI and 92% using agentic AI faced higher-than-expected costs. These widespread overruns indicate that current planning models do not align with actual workloads. Gartner anticipates that AI will enter the "trough of disillusionment," where the main barrier to scaling is the unpredictability of returns, not lack of ambition.
A November 2025 McKinsey survey of nearly 2,000 organizations in 105 countries found that while AI adoption is widespread, only about one-third are scaling it enterprise-wide, and just 5-6% report significant financial impact. The lesson is clear: unpredictable spending, without clear value, penalizes poor cost management.
Forecasting and governance remain challenging. A 2025 survey by Mavvrik and Benchmarkit found that 80% of companies miss their AI infrastructure forecasts by over 25%, and nearly 25% miss by more than 50%. Only 15% are within 10% of their actual spend. Additionally, 84% of firms report gross-margin erosion of 6% or more due to AI infrastructure. For software companies, these are recurring costs, not discretionary innovation spend. The key takeaway is that limited visibility makes AI costs difficult to manage, especially since only about a third of firms report on-premises AI expenses.
More considerations related to cost
Costs are not the only concern. As governments strengthen AI governance and standards, relying on externally owned models introduces risks related to access, terms, and availability. The practical issue is that a model not under company control is effectively a rented capability.
Currently, AI costs increase more rapidly than traditional technology expenses, often appear later, and do not follow predictable patterns. Minor changes to agent orchestration or prompts can significantly increase token consumption. Costs that cannot be attributed or managed cannot be fully controlled.
Capital and operating costs of AI
Two distinct costs of investing in AI systems need to be considered: capital and operational. Capital expenditure covers assets that a firm owns, such as chips, servers, storage, and networking equipment. These investments are often physical with a large upfront cost and depreciation over time. Operating expenditure includes rented or metered resources like cloud instances, tokens, power, cooling, and floor space. While these operating expenses are easier to initiate, they become significant at scale.
Key cost drivers that span both categories are: throughput (tokens per second), unit price (cost per million tokens), responsiveness (time to first token), and most importantly, utilization. Utilization is the primary factor in determining whether owning or renting is more cost-effective.
Reduce AI costs through efficient deployment
Hardware determines maximum capacity, but model efficiency dictates actual performance, and thus is a key variable for predicting cost. Cohere models range in their parameters and quantization, but approaches like routing tokens through a subset of model parameters (the mixture-of-experts or MoE method) lower generation costs without significant capability loss. Furthermore, models with low-bit quantization can increase throughput and reduce latency, maximizing hardware output.
For these reasons, model routing, or rather, choosing the right-sized model for a specific task, is important to reduce the cost for model inference. These techniques are essential for achieving cost-effective performance on owned hardware.
This is why raw hardware price alone is also a poor guide to cost. NVIDIA, whose chips do much of the world's inference, argues that the metric that matters is not compute cost or theoretical operations per dollar, which measure what you pay, or cost per token, which measures what you actually produce. By its own account, a cheaper GPU that delivers fewer tokens per second ends up costing more per token, not less. The generational gains are large: NVIDIA reports that its Blackwell-class systems deliver roughly 50 times more tokens per megawatt than the previous Hopper generation, cutting cost per token by about 35 times.
To make the trade-off concrete, picture a customer-support assistant that runs continuously: say, a half million requests a day, a few thousand tokens each, counting the retrieval and retries behind every answer. Rented through a frontier API, usage and costs never stop. Run the same workload on owned hardware and the cost structure inverts. The hardware becomes a fixed, depreciating expense, and each additional request after that is nearly free. Below a certain volume, the API wins because idle hardware is pure waste; above it, ownership pulls ahead and the gap widens with every token.
At the data center scale, the gap shows up in hard numbers. A 2026 Lenovo analysis was run on its own 8-GPU servers serving 70- to 405-billion-parameter models, a larger footprint than the single example above. It put the amortized cost of generating a million tokens at roughly $0.11 on owned H100 hardware, against about $0.89 for the equivalent cloud instance and around $2.00 for a comparable frontier API.
That is an eight-fold edge over rented cloud infrastructure and as much as eighteen-fold over buying tokens by the API, with five-year savings that run into the millions of dollars per server under sustained use. The same analysis found an 8-GPU server pays for itself in under four months — but only against on-demand cloud pricing, the most expensive way to rent.
Two independent estimates, reached by different methods, put owned inference in the same neighborhood. Lenovo's ~$0.11 comes from a five-year ownership model on H100s; NVIDIA reports $0.123 per million tokens on its flagship GB300 platform, measured by the third-party SemiAnalysis InferenceX benchmark. The figures are not strictly comparable — one is an accounting amortization, the other an inference benchmark on newer chips — but two sources landing near $0.11–0.12 for self-hosted output, against roughly $0.89 for a cloud instance and $2.00 for a frontier API, is a sturdier basis than either number alone.
Two things temper these figures. They are vendor costs, computed on that vendor's hardware, and the cloud side deliberately omits storage, egress, and support costs, so one could imagine the real spread is narrower than the headline. The same caution applies to the cost-per-token framing itself: it is NVIDIA's, and it conveniently showcases NVIDIA's own chips, with analysts noting the metric flatters highly optimized, homogeneous environments more than messy, real-world ones.
And every number assumes sustained, high utilization. The same analysis shows that owning an 8-GPU box turns out to be cheaper than renting one with only roughly four hours of use per day. The honest version of the argument is not that ownership always wins, but that it wins decisively for always-on workloads.
AI ownership in practice
In practice, ownership means owning or holding meaningful control over the value chain of components that an AI system depends on: data centers, chips, and the model layer, rather than paying another company for access to those things through an API.
A firm might own its models but rent the hardware, run open weights in a colocated facility, or buy all three as a single metered service. At the far end of dependence, the most strategic capability a business now relies on is one that it neither owns nor governs: someone else's model, on someone else's chips, in someone else's data center, billed by the token.
And as governments tighten AI rules and export controls, that dependence becomes a strategic exposure as much as a financial one. A capability that can be repriced, rate-limited, deprecated, or restricted by another company — or another country — is rented, however much is spent on it.
Ownership provides control over model deployment, safeguards data, prompts, and fine-tuned weights, and enables predictable, amortized costs instead of variable, usage-based expenses. Built around owned infrastructure, efficient models, and honest cost attribution, an AI strategy then becomes something that a board can plan around and defend, rather than an expense to brace for each quarter. The cloud still wins at what it was built for: bursts of training, experimentation, and unpredictable demand. But the workhorse inference that drives long-run cost increasingly belongs closer to home.
To connect with Cohere sales to see your total cost of AI ownership and discuss opportunities to lower it, please contact us.