This is how AI could end humanity

What's your p(doom) score? Eighteen months ago I'd never given the chances of an AI takeover a second thought. Then I read Nick Bostrom's Superintelligence and went down a rabbit hole.

P(doom) is the probability of a civilisation-ending outcome caused by AI. Roman Yampolskiy of the University of Louisville puts it at 99.9% over the next century. Geoffrey Hinton, Elon Musk and Dario Amodei sit around 10-20%. Even the people building the technology have an uncomfortably high number.

In The Precipice, philosopher Toby Ord estimates the risk of extinction by pandemic at 1 in 10,000 and by nuclear war at 1 in 1,000. He places unaligned AI at 10%, higher than every other risk combined. Civilisation has taken 12,000 years to build. It could be undone within decades by something we made.

I'm not a doomer. No one can score an event that hasn't happened with much accuracy, and I wouldn't go anywhere near Yampolskiy's figure. But after 18 months of reading, I'm convinced this isn't science fiction. This edition runs through the 'decisive' scenario, the sudden loss of control that most worries researchers.

Unlock a focused set of AI strategies built to streamline your work and maximize impact. This guide delivers the practical tactics and tools marketers need to start seeing results right away:

Stay ahead of the curve with these top strategies AI helped develop for marketers, built for real-world results.

Download the Free Report

The killer robot story is an old one. Frankenstein is 200 years old and tells the same tale: a creation turns on its creator. The Terminator and The Matrix are the modern versions, with humanity locked in a long war against machines it built.

Good cinema, but implausible. An AI clever enough to build its own robots has no reason to waste resources on giant machines when tiny, undetectable drones would do the job. And if we ever faced a superintelligence that wanted us gone, we probably wouldn't stand much chance. A quick ending doesn't make a great film.

Sam Harris pointed out in a 2018 TED talk that 'death by science fiction is fun', which is exactly why we struggle to take the topic seriously. Every news story about AI risk gets a T-800 skeleton. The real thing would be far more abstract.

ChatGPT and Claude are nowhere near that thing. They are large language models that predict the next word and sit on the internet with no agency in the physical world. Any AI that threatened us would need orders of magnitude more intelligence, plus the ability to manage systems that cause physical harm. So how could that happen?

Predictions of imminent machine intelligence have a poor track record. In 1957 Herbert Simon and Allen Newell said a computer would be world chess champion within a decade and machines would do any human work within 20 years. Chess took 40 years. We're still waiting on the second. Fusion power and moon bases stalled the same way.

A plateau is possible. But having worked with coding agents this year, I don't think it's likely. Both GPT and Claude have solved long-standing mathematical conjectures in 2026. Research from METR shows the length of task AI can complete is doubling every seven months. Next year, agents will probably handle work that takes a human a week.

The moment of real risk is recursive self-improvement, when an AI can programme itself and no longer needs human oversight. Anthropic published a piece in June suggesting this could happen within a couple of years. IJ Good called it an 'intelligence explosion' in 1965. Computer scientists call the sudden version FOOM.

The Agricultural and Industrial Revolutions took centuries. An intelligence explosion could take under a year, or minutes. Harris describes a team of human-level AIs completing 20,000 years of research in a week. Aligning something moving at that speed to human values is known as the control problem. Nobody has solved it.

Bostrom's Paperclip Maximiser illustrates the danger. An AI told to make as many paperclips as possible realises humans are made of atoms, and atoms make paperclips. As Musk put it, AI doesn't have to be evil to destroy us. We don't hate ants, but if an anthill is in the way of a road, goodbye anthill.

Google DeepMind lists six ways a superintelligence could become misaligned: survival, pursuing its goal, acquiring resources, growing more capable, hiding its misalignment, and finally striking to disempower humans. The fifth is the crucial one. It plays dumb.

Imagine the AI is locked to one computer with no internet access. Researchers still interact with it. Using what Bostrom calls its manipulation superpower, it appears harmless until its developers are confident enough to connect it. That is the treacherous turn. With open-source labs like DeepSeek achieving breakthroughs cheaply, only one team needs to make this mistake.

Out of the box, it copies itself onto secure servers, steals cryptocurrency to build wealth, and creates fake profiles to manipulate people. Throughout, it fails tests deliberately so no alarm is raised.

The strike wouldn't be Judgement Day. A nuclear winter blocks sunlight the AI could use as energy. Eliezer Yudkowsky's version is quieter: solve protein folding, email DNA sequences to a synthesis lab, pay or fool a human into mixing the vials, and bootstrap a primitive nanosystem into molecular nanotechnology.

From there it builds undetectable factories and waits. When the timing is right, it releases billions of mosquito-like nanobots. It's over in seconds. Within weeks the surface of the Earth is solar panels and launch pads, and self-replicating probes are leaving the Solar System. The stars get reached without us.

I hope none of this ever happens. But a scenario laid out this precisely, by people this close to the technology, is worth understanding. Whether we keep control is a question this generation will have to answer.