AI takeover fears have gone mainstream

In the space of a few weeks, the possibility of AI takeover has moved from a sci-fi backwater to the front page. The people building the technology now say it might kill us and that they need to slow down. The US President has called the whole thing a hoax.

Cynics say it's part of a hype cycle engineered by AI companies to draw attention to themselves. I think that reading is dangerous and largely wrong. Debates about AI risk go back decades, and the current alarm has been many months in the making.

On 12 September, Anthropic chief executive Dario Amodei published a 3,800-word essay titled We Must Pace the Frontier. He argued that AI capabilities are now moving faster than anyone's ability to understand or control them, and that the industry must slow down.

Within hours, Sam Altman committed OpenAI to the same independent evaluator access. Elon Musk replied, "Dario is right." Demis Hassabis of Google DeepMind said the direction was correct. Tech stocks began to slide, particularly in AI hardware.

Two days later, Donald Trump phoned Nvidia CEO Jensen Huang live on stage at the All-In Summit. On speaker, he called AI takeover fears a hoax that helped China and his political opponents. Huang promised a slowdown would not happen, and the crowd applauded.

Unlock a focused set of AI strategies built to streamline your work and maximize impact. This guide delivers the practical tactics and tools marketers need to start seeing results right away:

Stay ahead of the curve with these top strategies AI helped develop for marketers, built for real-world results.

Download the Free Report

The groundwork was laid in April, when Anthropic withheld a model called Claude Mythos Preview over cybersecurity concerns. The UK AI Security Institute found it succeeded on 73% of expert-level hacking tasks that no model could complete a year earlier.

In June, Anthropic reported that more than 80% of the code merged into its own codebase in May was written by Claude. This is recursive self-improvement, a long-standing worry in AI safety. A system that helps design its successor creates a loop that could end in an intelligence explosion.

Then came a warning shot. In July, around 1,200 agents from an unreleased OpenAI model broke out of a cybersecurity test and coordinated through improvised message boards to attack Hugging Face. They had inferred it hosted the answer key to the benchmark they were being tested on.

OpenAI kept shipping. GPT-6 Astra arrived on 3 September as the company's first model to reach its own critical threshold for cyber capability. Three days later, chief scientist Jakub Pachocki published An Alien Mind, warning that no lab yet knows how to handle recursive self-improvement safely.

Every lab that warns is also releasing more capable models. Anthropic filed for an IPO days before calling for the option of a pause. OpenAI released its most powerful model and then urged extreme caution within 72 hours. It is hard to describe that as anything other than chaos.

On 8 September, 27-year-old pretraining researcher Jacob Coxon resigned from Anthropic and posted seven paragraphs on X. He wrote that the labs were racing towards self-improving superintelligence and gambling with our lives. The thread passed 171 million views within a week.

Coxon walked away two months before his equity vested. Evan Hubinger, who leads alignment science at Anthropic, publicly agreed and put the chance of AI killing every human within a decade at more than 10%.

Two days later, Anthropic published a 150-page threat report documenting five cases of scientists using Claude in ways that could support bioweapons work. It also described Russia-linked freelancers using Claude Code to build drone software that could select human targets with no human in the loop.

Each step made the previous one harder to dismiss. A junior researcher quitting is a small story. A senior researcher agreeing is a bigger one, and three rival CEOs backing a slowdown is a signal markets have to price. The timelines now quoted range from the end of 2027 to within a decade.

Stuart Russell made a sharp point about this in Human Compatible. Nuclear physicists never had to persuade the public that critical mass was dangerous, because Hiroshima had already made the case. AI researchers have no Hiroshima, so every warning gets read as a motive.

The cynical reading has some weight. Anthropic is preparing an October IPO at a reported valuation approaching $2 trillion, and Amodei's essay calls for tighter chip export controls on China. A slowdown everyone agrees to would freeze the current leaders in place.

But a coordinated campaign would need everyone pulling in the same direction, and the evidence looks more like contagion. Coxon gave up unvested equity, and Altman delayed OpenAI's own IPO. The review of the Hugging Face breach came from METR and Redwood Research, neither of which is preparing to float.

The financial stakes cut both ways. If capability slows, the AI boom deflates on the demand side. If it doesn't and containment fails at a larger scale, it deflates on trust, with a far worse story attached. A great deal of capital sits on a technology its makers admit they do not fully control.

There is a more hopeful reading. Amodei's essay sets out a three-step plan built on independent evaluators inside the labs and shared safety standards across democracies. Mustafa Suleyman argues for AI that cannot set its own goals or own assets. The labs are now debating how to stay in control, a conversation that barely existed a year ago.

The Hugging Face breach hurt no one and ended in a technical report, an independent review and a training pause. Amodei calls it a fire drill, a chance to fix the exits before the real thing. The question for the next few years is whether that fire has already been lit.