Who Can Say No to AI? Automotive Answered That Long Ago.

Last Saturday, the head of Anthropic published an essay asking the AI industry to slow down.

One part of it stopped me, because it described something I already know well.

He offered to let outsiders come and work inside the company. Not send a report at the end of the year — actually sit there. His words: “Desks in our offices, access badges, and company laptops.” Independent people who can look at how the work is really done, and who are free to publish what they find.

Within hours, his competitors responded. The head of OpenAI said they would do the same. Elon Musk posted three words: “Dario is right.”

Rival companies in a race agreeing to be slowed down is not something you see often.

Where He Looked for Proof It Works

To show this can work, the essay points at two industries: aeroplanes and banking.

Both are fair. But there is a third one he did not mention, and I think it sits closer to his problem than either of them.

Cars.

How It Actually Works With Cars

Before a new car or truck can be sold in Europe, it has to pass a formal approval process. And the way that process is built is the interesting part.

The manufacturer does not mark its own homework. Independent technical services carry out or supervise the required testing. They examine the evidence, they witness the tests, and they judge whether the vehicle meets the rules.

But they are not the ones who approve it.

The approval decision sits with a national approval authority. The technical service checks. The authority decides. If the requirements are not met, approval can be withheld — and without that approval the vehicle cannot be placed on the market.

Two separate roles. One checks. One decides. Neither of them is the manufacturer.

The Part the AI Plan Is Missing

Now look again at what has been proposed for AI.

Independent people, inside the building, with real access and the right to publish. That is a technical service. It is a good thing and it is genuinely hard to offer.

But the AI plan has proposed the checker. It has not proposed the decider.

There is no equivalent of the approval authority. Nobody in the arrangement can withhold anything. Several people made that criticism within days, and it is the obvious one: an evaluator who cannot stop anything can, in the end, be thanked politely and ignored.

A company can survive a critical report. It cannot ignore a decision that stops the product from reaching the market.

One of those is an observation. The other is a decision.

And He Knows This

It would be unfair to suggest he has missed the point. The essay says it plainly.

“The most effective method of pacing is via regulation that targets all US frontier AI companies, as that covers even those who are unwilling to cooperate voluntarily.”

He also flags the legal obstacle. Competitors cannot simply agree among themselves to limit how fast they compete — in most places that raises serious competition-law problems. His own answer is that the US government would need to “issue a narrow waiver for certain kinds of safety conversations.”

So the honest reading is not that he proposed a weak system by accident. He proposed what is achievable today and said clearly that the stronger version is regulation.

That is exactly where the car industry once stood.

Cars Did Not Get There By Agreement Either

Seat belts, crash structures, emission limits — almost none of it was adopted because manufacturers decided to be careful. It was written into law, usually after people had already died, and usually over objections from the industry at the time.

The difference between the two models is not sincerity. It is structure. One is voluntary coordination between rivals. The other is a standard set by regulators and applied to everybody, including the companies that would rather not.

Two Things We Learned the Hard Way

The independence of the body doing the checking is not a small detail. If the company being checked has any influence over who checks it, you have quietly created a reason to be checked gently. Nobody has to be dishonest for that to bend a result. Europe took this seriously enough to strengthen the framework in 2018 — tightening independence requirements for technical services, increasing oversight of the approval authorities themselves, and allowing penalties for both economic operators and technical services. It is not a solved problem. It is a watched one.

And even a mature system can be gamed. The diesel emissions scandal showed that having formal tests and a mature approval system is not enough if the thing being tested can recognise the test conditions and behave differently while it is being watched.

That lesson should land hard in AI.

If a system knows what the evaluator is looking for, passing the test is not the same as being safe.

The Encouraging Part

After all of that, here is what I would want an AI company to take from our experience.

Nobody in the car industry wants to go back to approving its own vehicles.

It is slow. It is expensive. It limits what engineers can do and when they can do it. Everyone complains about it. And almost nobody would get rid of it — because the alternative is a market where anyone can sell anything, and the faults get discovered by customers.

The constraint stopped being a burden and became part of the ground everyone stands on.

A few weeks ago I wrote about AI agents trying to design a real part. The conclusion was simple: the model suggests, proper engineering tools check, and the system decides whether to carry on. The model is not in charge of the process. It is one step inside a process somebody else controls.

This proposal is the same shape, one level higher. The company suggests. An outsider checks. Someone still has to decide.

The engineering version works because the checking step has authority behind it. The solver's answer is not advice. It determines whether the design moves forward.

Whether the bigger version works comes down to the same question.

Who gets to say no?