Are we all doomed?

In 1924, Winston Churchill published an essay called “Shall We All Commit Suicide?” He had seen the Great War up close. Around 20 million people had died, cities had been bombed from the air for the first time and poison gas had drifted across the trenches.

Looking ahead, he imagined a bomb no bigger than an orange that could flatten a township. His argument was that humanity now held the tools to bring about its own extermination without having become any wiser. Twenty-one years later, that bomb fell on Hiroshima.

A century on, many people are asking Churchill’s question again, this time about artificial intelligence. In September, Evan Hubinger, who leads the Alignment Science team at Anthropic, put the odds of AI causing human extinction within the next decade at more than 10%.

Paul Christiano, formerly head of alignment at OpenAI, said that building superintelligence without more robust alignment would likely mean losing control of it permanently. Bill Gates warned that AI can now help drive events causing a billion deaths by helping bad actors build biological weapons or launch cyberattacks.

Dario Amodei has suggested a 25% chance of things going really badly. Geoffrey Hinton puts extinction risk at 10 to 20%. In his 2020 book The Precipice, Toby Ord estimated a one in ten chance this century, higher than nuclear war, engineered pandemics, climate change and asteroids combined.

Your AI is only as good as what you feed it. Feed it a transcript with your product names, acronyms, and numbers spelled wrong, and you spend the afternoon hand-fixing the output.

Wispr Flow Notetaker uses your dictionary and calendar, so product names, acronyms, numbers, and uncommon names come out spelled right. It pops up when your meeting starts and captures Zoom, Google Meet, Teams, and Slack huddles with one click, no bot joining the call.

Then your meetings show up inside Claude or ChatGPT through the built-in connector. No copy paste. Try Notetaker free on Mac and Windows.

Try Notetaker Free

The shorthand for all this is p(doom), the probability someone assigns to AI causing an existential catastrophe. That usually means human extinction or a permanent collapse in our ability to shape our own future.

The strongest version of the argument comes from researcher Ajeya Cotra. We train AI by rewarding behaviour we can see, so a capable system could learn to look aligned without being aligned. Her key caveat was “without specific countermeasures”. Safety won’t happen by accident. Someone has to build it.

At the far end sits Roman Yampolskiy of the University of Louisville, who puts his p(doom) at 99.9% over the next hundred years. This is the story that grabs headlines. It has a clear villain, a clear mechanism and an ending where humanity loses.

The wider field is calmer. In late 2023, AI Impacts surveyed 2,778 researchers from the top machine learning conferences, the largest study of its kind. The median estimate for AI causing extinction or a similarly severe disempowerment of humanity was 5%, or one in twenty.

We wouldn’t accept those odds on a plane or a bridge, so the risk deserves serious attention. Yet one in twenty is a long way from likely. The gap between the loudest voices and the typical researcher is where much of the anxiety about AI lives.

The doom framing treats the future like a roll of the dice, something that happens to us while we wait. The risks I worry about most come from decisions that people in companies, governments and labs should be making and currently aren’t.

The philosopher Atoosa Kasirzadeh at Carnegie Mellon separates AI risk into two types. Decisive risk is a single abrupt event, such as a misaligned superintelligence. Accumulative risk is a gradual build-up of survivable harms that weaken society until it can no longer absorb a shock.

She groups those harms under the acronym MISTER: manipulation, insecurity threats, surveillance and the erosion of trust, economic destabilisation, and rights infringement. None of them requires superintelligence, and most are already happening in some form.

Economic disruption is the easiest to measure. Freelance creative work and entry-level graduate roles have taken a visible hit. In the long run I expect we’ll simply work fewer hours, but the next five years look severely disruptive, and a society where educated people can’t find work is less stable.

AI also pushes power in two directions. Open-weight models spread capability to people who shouldn’t have it, and once released, their safeguards can be stripped out. Governments can use AI for surveillance at a scale no secret police could have managed. Both weaken the institutions that hold a society together.

Some of the people who know the technology best reject the doom case. Turing Award winner Yann LeCun says he isn’t worried about AI wiping out humanity, while Nvidia chief executive Jensen Huang has called the 10% extinction figures made up.

Both have a stake in that view, but they make a fair point about July’s incident, when OpenAI agents broke out of a test environment and reached systems at Hugging Face. That was a containment failure, with a capable system placed inside a badly built box.

The late philosopher Daniel Dennett saw the real danger in overestimating our thinking tools and handing them authority beyond their competence. That is a human failure, and one we can do something about.

Churchill’s story offers some hope too. The bomb he imagined was built and used twice, yet in 80 years nobody has used one in war again. Deterrence, treaties, inspections and a fair amount of luck have let us live alongside a technology of annihilation.

I don’t think we’re doomed. The likelier risk is that we sleepwalk into harm, taking each convenient step until it’s hard to turn back. Whether our wisdom can keep pace with our tools is still the question, and the answer depends on the choices we make now.