Dario Amodei’s “Pacing Proposal” calls on AI developers to slow the training and release of new models and to embed independent evaluators within their own firms as permanent auditors. Sam Altman and Elon Musk have immediately expressed support on social media.
Amodei wants governments to make this compulsory. If lawmakers act, rules written over the next year or so will determine who is allowed to build advanced AI for the next decade. Meanwhile, the public is asked to accept a safety judgment based on internal evidence that perhaps a few dozen people have seen. There are too many unknowns to assess whether pacing can work.
AI externality
Economists usually think about negative externalities through pollution. A factory dumps waste into a river. It keeps the profit and offloads the cost onto the town downstream. Because the firm never feels the harm, it under-invests in preventing it. That is why we regulate such “market failures”.
It seems Amodei insinuates that AI externality is different because a developer that releases a dangerous system is somewhere in the blast radius too. But what disciplines behaviour is not this absolute apocalyptic scenario. You cannot fine a company in a world that no longer exists. What disciplines it is something far more ordinary — being the named defendant in a lawsuit when a model enables large-scale fraud, a serious cyber incident, or an agentic system that hurts real people.
In this more ordinary scenario, pacing is not civilisational insurance, as it is pitched. It is simply risk management ahead of a liability regime nobody has written yet. That is a reasonable thing for a company to do. It is just a different thing from what the framing suggests.
Two worlds, one announcement
Suppose the returns to building ever-larger models are already flattening, and the people inside know it. Then pacing is free. You announce a restraint you were going to face anyway and collect the credit, akin to OPEC announcing production cuts amid falling demand. Now suppose the opposite: internal testing has found something very alarming. These are very different worlds, but they produce very similar announcements, perhaps with a slight difference in tone. We don’t know.
Who joins, and what follows
A speed limit only binds whoever is speeding. A lab six months behind the frontier can sign up at no cost and freeze its deficit in place. A company already selling itself on safety is paying most of the cost anyway. So, the firms most likely to join are the ones the agreement would not have restrained, and we don’t know who joins for what.
The deeper effect is on who never joins at all. Industries tend to acquire the regulation they want because the benefits concentrate on a few firms while the costs spread across everyone else, a mechanism known as regulatory capture. Here the mechanism is simple. Embedding auditors creates a baseline compliance cost. For a company that already runs safety, policy and compliance teams, the extra burden is small. But for a startup in its first year, the same requirement could raise the cost of entering or competing at the frontier.
What game theory says
Two games in economics are useful to describe the situation. First, in a stag hunt game, two hunters together can bring down a stag, in which case both eat well. Alone, each catches a hare and goes hungry, but not starving. The stag only works if the other hunter commits too, and if he does, you have no reason to defect. Cooperation holds itself together.
The other game is the prisoner’s dilemma, where racing ahead is your best move no matter what anyone else chooses, and only outside enforcement sustains cooperation. That is the game Amodei appears to think we are in, which is why he wants governments involved.
There is another reason to want governments involved. An agreement among competitors to restrict output can raise collusion concerns. Voluntary pacing has no legal cover, but a mandate does. So, the call for regulation solves two problems at once.
What the markets are told
Anthropic is preparing for what is expected to be a very large public listing. Sam Altman has said OpenAI will not go public this year, citing concerns about AI safety.
Both are consistent with pacing being cheap. Regulatory uncertainty is a valuation discount, so a rule you helped write and already meet is a rule the market can price. And “we are pacing” is a dignified way to slow capital spending without conceding to investors that the returns to scale are falling.
But both are also consistent with the opposite. Delaying a float costs real money, and rivals may list first. A company using safety merely for public justification would not pay that price.
What we need to know
Whatever else is going on, we don’t know. What we do know is that there are at least three questions that need to be addressed before we can take pacing seriously, or ask how to make it work.
- What would falsify the claim that pacing is costly? Without an answer, we have no way to distinguish a costly constraint from one that was never binding.
- What is the counterfactual? Without knowing what would have happened without pacing, we cannot measure its effect on the path of AI development.
- What is the cost of pacing, and what is the benefit? We need some way to weigh both sides of the ledger and understand the trade-off.
The costs and benefits may not fall in the same place. Domestic pacing may impose costs locally while its benefits, if any, are global. But if a rival country continues to develop rapidly, the country that paces bears much of the cost while the global trajectory may barely change. Moreover, beyond the seemingly “straightforward” costs of pacing, such as delayed medical research and productivity gains, and potential benefits such as reduced AI-related risks, there is a more difficult informational trade-off. Without pacing, we learn by doing — failures reveal vulnerabilities, firms and regulators learn what goes wrong, and safety techniques improve. With pacing, we give up some of that learning-by-doing, but gain time to learn before committing further. Which kind of information is more valuable, and how do we know?
We need an understanding of what we are buying, what we are giving up, and only then think about which mechanism actually gets us there.
Nihad Aliyev, Isa Hafalir, Ali Furkan Kalay