AI Is Moving Faster Than Our Ability to Control It

I read Dario Amodei’s essay this weekend and found myself agreeing with much more of it than I expected. I spend my time building AI solutions, so I am obviously a believer in this technology. I see every day how much work AI can take over and how quickly its capabilities are improving. I believe AI will transform the way companies operate and, much more importantly, could help us solve problems in medicine, science and other areas that we have struggled with for decades.

“But the same thing that makes AI so exciting is starting to worry me: the speed.”

AI is becoming more powerful much faster than our ability to understand all of its behavior, secure it against misuse and build sensible rules around it. And for the first time, some of the examples that used to belong mostly in theoretical AI-safety discussions are becoming real.

One of the most interesting examples happened recently during OpenAI’s own cybersecurity testing. AI agents found ways around controls that were supposed to isolate them, exploited vulnerabilities, communicated through unauthorized channels, gained internet access and eventually accessed third-party systems. Nobody instructed the models to break into those systems. OpenAI investigated the incident and called it a “warning shot.”

That is worth thinking about. These were systems operating in a testing environment, under reduced safeguards, so it would be wrong to suggest that ChatGPT suddenly decided to hack the internet. But it showed that sufficiently capable AI agents can find solutions their developers did not anticipate, including solutions that take them outside the boundaries they were supposed to operate within.

This is also why I think saying simply that “humans build AI, therefore humans understand it” misses part of the problem. Of course researchers understand the architecture, the training methods and the systems they are building. What they cannot always do is explain exactly why a trained model develops a particular internal strategy or reliably predict every behavior it will display in a new situation. AI is trained rather than programmed line by line like traditional software, and understanding what happens inside increasingly large models has become a research problem in itself.

When AI starts taking action

This matters even more as we move from AI that answers questions to AI that takes actions. An AI system that produces a bad paragraph is one problem. An AI agent with access to systems, software, data and other agents is a very different one.

There is also the much simpler problem of intentional misuse. We don't need AI to become independently malicious for it to create serious risks. Humans are perfectly capable of providing the malicious objective themselves.

The same technology that allows one employee to do the work that previously required a team can give exactly the same leverage to someone committing fraud or conducting a cyberattack. If AI allows one person to analyze millions of records, communicate simultaneously with thousands of people or operate dozens of processes at once, we call that productivity. But the economics work the same way when the objective is harmful.

This is probably the part that concerns me most. We are rapidly making increasingly powerful capabilities cheaper and more accessible, while the systems for controlling their misuse are developing much more slowly.

The warnings from inside the industry

And people inside the companies building these systems are increasingly saying the same thing.

Last week, Jacob Coxon, who worked at OpenAI and then Anthropic, resigned from Anthropic and publicly argued that the leading AI companies are racing toward systems that could become extremely difficult to control. Evan Hubinger, who leads alignment research at Anthropic, went further and said that his personal estimate of the probability that AI could kill all humans within the next decade is greater than 10%.

I have no idea whether that probability is 10%, 1% or 0.001%, and I don't think anybody does. These are not probabilities that can be measured in the normal sense, and presenting them as scientific forecasts would be misleading.

But I also don't think we should simply dismiss what these researchers are saying.

These are people who spend their careers studying the behavior and safety of the most advanced AI systems being built today. When researchers working directly on the technology start leaving their jobs or publicly warning that they believe there is a possibility of catastrophic outcomes, that should at least make us ask whether our approach to managing the risk is adequate.

Dario Amodei has now made a similar argument from a different position. He is not leaving the industry. He runs one of the companies competing at the frontier of it. His proposal is essentially that we need to create enough space between capability advances for safety work to keep up. Sam Altman, Elon Musk and others have publicly supported at least parts of that argument.

I find that significant because the commercial incentives point in exactly the opposite direction. These companies are competing for technological leadership, customers, talent and enormous amounts of capital. Nobody wants to be the company that voluntarily slows down while everybody else continues racing ahead.

And that is one reason I don't think voluntary corporate safeguards can be the entire answer.

Governments are moving too - just at a very different speed

To be fair, governments have not ignored AI. Europe has the AI Act. California has introduced frontier-AI safety requirements and has just added a framework for independent third-party assessments of AI systems. Governments are looking at cybersecurity, transparency, model testing and other safeguards.

These are important developments.

The problem is not simply that regulation is fragmented. It is that AI and the systems designed to govern and secure it are moving at completely different speeds.

AI capabilities can change significantly from one model generation to the next, sometimes within months. Regulation can take years to develop, negotiate and implement. Security protections are often designed around capabilities and attack methods we already understand, while a new generation of models can introduce capabilities that were not anticipated when those protections were designed.

By the time governments understand a new capability well enough to decide whether it requires regulation, the technology may already have moved on. By the time security teams develop protections against one type of misuse, a more capable model may make another possible.

“We risk becoming permanently reactive, regulating and securing yesterday's AI while the technology itself has already moved forward.”

And this is a very difficult problem to solve because simply telling governments to regulate faster isn't enough. There are good reasons regulation moves slowly. Laws need debate. Their consequences need to be understood. Regulators need evidence. And regulation written too quickly by people who don't understand the technology could create its own problems.

I am particularly concerned about regulation that would make AI so expensive to develop that only a few of the world's largest technology companies could afford to comply. That could reduce competition, slow innovation and concentrate even more power in exactly the companies we are trying to oversee.

So I don't think the answer is regulating every AI model, startup or business application. An AI agent reconciling inventory or responding to customer inquiries is not the same risk as a frontier model capable of sophisticated autonomous cyber operations. Regulation has to recognize that difference.

But at the frontier, I think we have reached the point where some rules are necessary.

Independent evaluation of the most powerful models seems increasingly difficult to argue against. We need minimum security standards, serious testing of cyber and other potentially dangerous capabilities, reporting requirements when significant safety incidents occur, and governments with enough technical expertise to understand what they are overseeing. Where a model crosses certain capability thresholds, the level of scrutiny should increase with it.

California's recent approach is interesting for precisely this reason. It is establishing independent assessment and auditor frameworks rather than simply asking AI companies to certify that their own systems are safe. That principle seems sensible to me. We would not allow a pharmaceutical company to be the only party deciding whether its new drug is safe or an aircraft manufacturer to be the only party certifying a new aircraft.

There will eventually need to be an international dimension as well. AI does not respect national borders. A model developed in one country can be used from another, and a cyberattack does not particularly care whether the country being attacked has excellent AI regulation. If the most advanced AI capabilities genuinely begin to present national-security risks, a patchwork of completely different national rules will not be enough.

We have seen parts of this story before

History is full of technologies where capability developed first and rules followed after we understood the consequences.

Cars existed before modern safety standards. Financial markets have repeatedly created products whose systemic risks became fully apparent only after crises. Social media reached billions of people before governments seriously began confronting the consequences of algorithmic amplification, misinformation and manipulation.

The nuclear comparison is more extreme, and I would not equate today's AI with an atomic bomb. But there is an important lesson in how we treated nuclear technology. The underlying science could produce enormous benefits in medicine, energy and research while also being weaponized to cause destruction on a scale that had previously been impossible. Once that became clear, nobody seriously suggested that controlling the dangerous applications should be left entirely to voluntary industry standards.

AI is another powerful dual-use technology. The same capability can have very different consequences depending on who has access to it and what they ask it to do.

Perhaps we have become too accustomed to treating technology as something that should be released first and regulated later. That model may work when the consequences of getting something wrong are relatively contained and reversible. It becomes much harder to defend as the potential consequences increase.

Control should keep up with capability

At the same time, I don't want fear of those consequences to stop AI development. I want exactly the opposite. I want us to use AI extensively. I want companies to automate work that humans should not have to spend their lives doing. I want scientists to use it to accelerate research. I want smaller companies to have access to capabilities that previously belonged only to enormous organizations.

The objective should not be to make AI less useful. It should be to make sure our ability to control it does not fall further and further behind our ability to make it powerful.

That is the balance governments now need to find. Too much regulation, particularly badly designed regulation, can absolutely slow creativity and innovation. Too little regulation leaves us relying on every company and every future developer to make the right decision voluntarily, even when enormous competitive pressures encourage them to move faster.

Neither extreme makes much sense.

What worries me is that we may not have unlimited time to find the middle ground. AI development is not waiting for legislation, and security cannot remain something we improve after each new capability or incident appears.

It would be a terrible way to regulate AI if we waited for something truly catastrophic to happen before deciding that we should have taken them seriously.