A perspective from Aidan Gomez, Co-founder & CEO of Cohere
Artificial intelligence is remaking the world we live in. Within a generation, the way we discover medicine, manage power grids, and secure our national infrastructure will be completely transformed. Many people already realize this and are working to build that future responsibly. Many people underestimate the scale and pace of change coming. Some, however, claim to foresee this change and use it to serve their own ends.
Here is the question nobody is asking clearly enough: should a handful of select, market-dominant AI companies from Silicon Valley get to define the rules and safety standards of a generational technology for the entire world? All while simultaneously determining how fast this technology progresses? We have tried that before with very poor results. Once again using fear under the pretext of protecting the public, these oligopolies are now requesting to bend competition rules and be permitted to dictate the terms for everyone else. A sheep in wolf's clothing, a cartel by any other name.
I believe in the potential of AI technologies to bring benefits to our world, and I do not downplay the risks. I run a company that builds AI systems deployed inside banks, telecommunications networks, and defense ministries. These are among the most high-stakes environments because failures in these sectors can have consequences far beyond an individual user – disrupting financial systems, critical infrastructure, national security, and essential services at scale. The same capabilities that find vulnerabilities in your code can find them in someone else's, and cyber offense is getting cheaper faster than defenses are getting better. That gap should worry you as much as it worries us.
AI needs guardrails. That is not the dispute and never has been. The dispute is over who writes them, who gets to participate and whose interests the rules are protecting. The question is truly about whether we should have the freedom to choose based on scientific evidence or if we should hand the reins of the most consequential technology in human existence to a few Silicon Valley executives.
We’ve been here before
Before I break down the self-serving framework the big labs are pushing and offer ideas for an alternative, let’s take a couple lessons from the recent past and think about the word cartel, because the history is specific and it is the accurate term.
In 1975 the Securities and Exchange Commission needed reliable bond ratings for its capital rules. It designated three firms as Nationally Recognized Statistical Rating Organizations and never published criteria for how anyone else might earn the designation. These were government-blessed outside evaluators, paid by the very issuers whose securities they graded, sitting behind a barrier the regulator itself had built. Twenty five years later there were still only three of these evaluators. Then they rated subprime mortgage securities triple-A and nearly took the global economy down with them.
Europe ran the experiment again in 1985. Car manufacturers lobbied for a sweeping antitrust waiver, the Motor Vehicle Block Exemption, arguing that modern vehicles were complex, safety-critical machines and that manufacturers therefore needed control over who was qualified to sell and service them. The subsequent regulation let manufacturers set the standards for premises, equipment and staff training, explicitly in the interest of safe and reliable vehicles. But what followed wasn't safer cars. It took the European Commission roughly twenty five years of reforms to unwind, and to establish what should have been obvious at the start: it is possible you can hold strict safety standards without handing the incumbents a monopoly on meeting them.
Nobody set out to build a cartel in either case. In both cases, the stated goal was safety. But the result was a market structure that protected incumbents and limited competition, all under the justification of serving the public interest. I don't doubt the sincerity of the people involved: many were concerned about the risks and worked earnestly to resolve them. But complex problems aren’t always solved on the first try, and any responsible scientist, engineer, or lawmaker knows that to solve new problems you must learn from past history.
What's Being Proposed
This brings us to the roadmap published this week by Anthropic CEO Dario Amodei, asking governments for antitrust exemptions in the name of safety. This roadmap is the latest in a string of recent efforts by Silicon Valley incumbents to shape the regulatory landscape surrounding AI.
I want to be clear about what we agree with. Independent review of highly capable AI systems is a good idea and we support it. However, many aspects of the proposal raise fundamental questions: who writes the standard those reviewers apply? Who conducts or oversees the review? Who gets to participate in the conversation that sets the rules?
On these questions, the proposal is clear. A handful of the most powerful labs based in one country would agree on shared standards and the limits to how fast the technology should advance. And here’s the key point: because it’s normally illegal for competitors to agree to limit what they produce, the plan asks governments for a narrow antitrust waiver to make that coordination lawful. And it also asks governments to require every other AI developer to blindly follow whatever the participants settle on – despite those other developers and wider society not having an opportunity to voice the impact or share their perspective on the science.
This is not a question of adding one or two more companies into the conversation. Adding an extra chair fundamentally doesn’t solve the issue. The problem is that there is a list at all, when the decisions being made reach every company, every government, and every citizen who never got asked. You cannot have it both ways. If this is the most consequential technology in human history, then the rules for it cannot be written by a small group of commercially aligned companies behind an antitrust waiver. There is no public comment period here. There is no consultation, and there is no vote. The public will be forced to live with the outcome regardless.
A safety regime designed by a few labs will only be rigorous about the risks they have already built their safety systems to assess and completely quiet about everything else, further entrenching their market position and limiting competition. Risk in these existing frameworks gets defined as a function of scale, which makes the companies with enormous systems the only ones qualified to judge. The types of risk deemed relevant for assessment are also pre-ordained, rather than up for scientific debate and alignment. For example, there is real disagreement in the field about how much offensive capability comes from a raw model size versus the harness wrapped around it. Smaller models orchestrated well, using tools and verification steps, can do things that large models can’t. A cyber swarm is a completely different risk surface than a single model. None of that shows up in a regime built exclusively around massive compute thresholds.
There's a sentence in the essay that any competition authority would find troubling. It promises that a coordinated approach would give developers time to do this safety work without sacrificing commercial advantage. But to whose advantage? The firms drafting the framework are the firms sitting at the top of the market today. A mechanism that slows everyone down while explicitly preserving existing commercial advantage does not make AI safer. It risks entrenching today’s dominant AI companies by turning their current advantages into baseline for what it takes to compete safely. Safety rules should reduce risk without regard to who leads the market or who stands to gain from the rules.
The entry requirements set out in the proposal tell you the rest. Vast computing power. Continuous monitoring infrastructure. Dedicated security organizations. Resident evaluator teams with desks and badges. Shutdown architecture. Government relationships that are deep enough to navigate all of it. A pool of “independent” evaluators that is already remarkably small, funded by the same handful of organizations repeatedly relied upon by the same frontier labs.
Convince a government that AI is an existential threat and you can convince it to outlaw your competition. The intention is clear and it does not create a safer world.
What Better Rules Look Like
So what will enable safe, responsible AI development? To be clear, I don’t believe I have all the answers - nor do I think I should get to make the rules instead. Rather, I will try to propose practical and effective ideas that can be considered alongside those of many others by governments and lawmakers as they use their democratic powers to set the direction of travel.
Those ideas are built on four pillars:
- An evidence-based risk framework. First things first, and before anyone mandates testing or auditing, we need an agreed and published account of which harms we are concerned about, which AI capabilities cause which harms, under what conditions and in what contexts, and at what point a government should step in. That account must be built across all the countries developing this technology, and in the open rather than behind closed doors under the banner of national security. Establish a coordinated, international effort to develop this framework that is not led by any one nation, but a group of them. Put technologists in the room next to the policy experts and experts from critical sectors like finance and critical infrastructure. Include researchers and scientists who disagree with each other and publish the disagreements, because an honest process shows its arguments instead of announcing its conclusions. Fund the testing capacity itself through public research bodies and existing sectoral risk management systems, so the science doesn't depend on the budgets of the companies being measured. And write rules that bind based on what an AI system can do rather than on who built it, so a dangerous capability is treated the same whether it comes out of a trillion dollar lab or a university department. The science of AI-related risk cannot and should not be divorced from the decisions companies and governments make about how AI is used and deployed, nor from existing and robust risk management systems that govern critical sectors today like healthcare, global financial systems, defense, and critical infrastructure.
- Mandatory transparency. AI developers should be transparent about how their models and systems are built, their intended purpose and capabilities, what risks they might pose, and what risk mitigation measures have been implemented. Model cards are already widely published across the industry for generative AI models deployed at scale, covering what tests were run and how the model performed. But more can be done, particularly around how companies across the development and deployment stack report serious incidents over the layers where they have visibility and control, and mechanisms to attach real accountability when real harm occurs.
- Testing, scoped by the evidence. The most advanced AI models and systems should face independent testing, but only against the capabilities and in the contexts the risk framework has identified as genuinely dangerous, rather than leaving that definition to a select few companies. In practice that likely means the ability to generate cyberattacks, synthetic fraud and voice cloning, manipulation at scale, physical or biochemical weapons, and anything touching critical infrastructure. It does not mean testing every system for every risk, and it must not become a compliance exercise that expands to fill whatever budget the largest firms can absorb. A tiered and proportionate framework where more-capable models and systems, or models or systems deployed in specific contexts, face more stringent testing - regardless of the resources put into developing them, will do the most for improving safety. Test what can harm people and societies, and let evidence decide what requires testing rather than whoever holds the pen. Certification has to be open to every company rather than restricted to a designated tier of AI developers, and the standard has to be agreed by people other than the companies being measured against it. What this can't be allowed to become is an expensive bureaucracy that chokes off smaller labs before they ever ship anything, which is exactly what happens when the scope is unlimited and the incumbents are the ones setting it.
- Real assurance mechanisms. The parameters that determine how AI models and systems are tested and the mechanisms that verify those tests must be truly independent, similar to the way financial institutions are licensed, aviation companies maintain strict safety standards, and nuclear facilities accept inspection. Such high-stakes industries already rely on layered assurance: developers test their systems, customers validate them against their own risk requirements, independent third parties provide additional assurance where necessary, and regulators oversee the framework. AI should build on these tried and tested approaches, rather than claiming unprecedented exceptionalism and assuming safety depends on a single class of permanently embedded evaluators. Assurance works when three conditions hold: one, testing and verification is based on collectively developed and published criteria; two, any involved third parties must have a mandate to include an array of opinions and never be paid by the party they're reviewing; and three, findings must reach the public in some way that isn’t conflicted. Most important is flexibility around which aspects of assurance work are best done in-house to strict standards and which require a third party, based on the criticality of the audit and the most efficient use of expertise and resources. This stands in direct contrast to what has been proposed: an assurance system based on auditors who not only have financial or ideological conflicts of interest with those they audit, but who are handpicked by them. Suggestions to give the auditors preferred by a handful of dominant companies continuous access across the industry are a path to regulatory and ideological capture, not safety or trust.
What the Panic Leaves Out
These pillars are the foundation of what a practical, risk-based approach looks like. Now compare it to the science fiction scenarios currently being weaponized by the largest incumbents.
I believe talking about science here is extremely important. There are a lot of logical leaps and conclusions being made by smart people. But it is important that rather than hand-waving, we discuss what they are, and what it means.
Earlier this week, a researcher quit a large lab with loud warnings that superintelligent systems will probably wipe out humanity within a decade. A senior colleague publicly chimed in to say he puts the odds above ten percent. I do not doubt their concerns are well-meaning and genuine. But let's remember those numbers didn't come from any fundamental reality. They are gut feelings, vibes, expressed as decimals, amplified by executives with vested interests and covered by the media for a week as though they were mathematical analyses.
It is worth being precise about what the worry actually is: as these systems get more capable, the distance between what we asked for and what we actually get becomes harder to notice and more expensive when we miss it. A system that is better at finding loopholes is also better at finding the loopholes we never thought to check for. Give it tools that act in the world, and a rate of improvement that outpaces our ability to review its work, and you can imagine catching problems long after it mattered, rather than right out of the gates.
However, the claim that such problems mean these tools are out of our control is a judgement call, not a finding.
Yet that distinction is the whole difference between science and science fiction, and it decides what we should do next. An open question of this kind is exactly what a public, contested, evidence-based process exists to work through. What you should never do with an open question is hand the people holding one particular view the authority to write binding rules from it and impose them on everyone else.
The failures we saw reported in July happened inside the two best-resourced labs in the world, with the largest safety teams, the most internal review, and in one case an outside evaluator arrangement was already being stood up through METR with a substantial effort as recently as February. The proposed remedy is more or less what was in place when it broke. A capability threshold would not have caught it, because those systems were actively being trained and evaluated to assess their capability. A compute limit might have slowed down the agents, but not reduce their capabilities. What failed was the quality of the instructions, and the strength of the walls around the test, and how long agents were allowed to continue working without observation. The proposal addresses none of these.
What would help is far less dramatic. Require that serious incidents be reported, so a flawed training setup at one company becomes a lesson for the whole field rather than a paragraph in a blog post. Test systems against the specific gaps that are known to get exploited. Ensure there are standards for test-time observability (or at least logging) to make sure bad behaviors are detected earlier. Insist that anything wired into critical infrastructure, from a hospital to an electrical substation be walled off, ideally on-prem, so that a system chasing a badly written score cannot reach anything that matters. And apply all of it according to where a system is deployed and what it can touch, rather than how large the company that built it is. A small, poorly specified model sitting inside a hospital is a live risk today, and under a frontier-only regime nobody is even looking at it.
When narratives that serve Big Tech interests take events like this and focus public attention on the idea of super-powerful, uncontrollable technologies that may lead to human extinction, it conveniently distracts us from the choices and mistakes they are making, and the harm experienced by real people right now. For example, voice cloning tools cheaper than a phone bill can empty a pensioner's account in minutes. Similarly, automated decision-making systems can have a real impact on access to essential services. We need a safety regime built for those realities, issues which directly impact citizens and organizations today, not one designed to contain a hypothetical superintelligence.
Who Writes the Rules?
We're at a turning point, and the decisions made over the next few months will shape the global economy for a generation. The risks are real and they need serious, enforceable safeguards. That's exactly why the rules can't be drafted behind a waiver by the companies they're meant to govern. The idea that two or three Silicon Valley companies should act as the creator, gatekeeper, and rulemaker for AI for every government on earth doesn't survive being said out loud.
Critical infrastructure cannot be secured by renting national capability from a foreign monopoly behind a closed interface. Hospitals, payment networks, and defense ministries, cannot, in good faith, pipe their most sensitive operational data and proprietary knowledge to somebody else's servers and blindly trust a vendor contract to hold. Loopholes enabling data leakage are already being exploited today.
We built Cohere precisely because of this reality. As a global business working closely with governments all across the world, we see what the institutions keeping these economies running actually need. They want highly capable systems running inside their own walls, operating on infrastructure they control, from providers who answer to them and can be replaced. Security comes from sovereignty, local deployment, and technological diversity. A competitive market with many capable suppliers can absorb a failure at one of them. A state-sanctioned cartel has nowhere to hide one.
The rules around AI are getting written either way. What's still open is whether they get written by a group anyone can join and with evidence anyone can check, or by a handful of companies in a room with the door shut. More voices makes it slower. It makes it harder. Some of those voices will say things the rest of us don't want to hear. That's the point. It's the only version that produces a rulebook the public has any reason to trust.
A process worth having would include people who'd rule against even companies like Cohere. Academics with no commercial stake. Civil society groups who think everyone in this industry is moving too fast. Smaller labs and open-source developers. Governments with their own reasons not to take our word for it. If we agree this technology is remaking the world we live in, a handful of CEOs and groups that they pay cannot be making all the decisions for how this technology evolves. We need more voices at the table.
You can talk about safety, slowing the pace to ensure progress is sustainable, and the need for others to step in to ensure you do things responsibly. Or you can just get on and do it: build responsibly and at a pace that is sustainable for society, enable sovereignty for your partners, work with and listen to lawmakers and safety experts. At Cohere, we’re choosing to do the latter.