In partnership with
In 2023 I read The Coming Wave by Mustafa Suleyman, the DeepMind co-founder who now runs Microsoft AI. He argues that AI and synthetic biology together carry more power than any technology in history, with a huge upside and problems on the same scale. Three years on, that dilemma is playing out in real time.
I've long thought AI needs proper regulation. There are more rules for running a sandwich shop than for building an AI model. Yet the US and China are racing each other, and I often feel fairly powerless to do much about it. Joseph Miller is one of the people trying to change that.
He leads PauseAI UK, part of an international movement pressing governments on the risks of superhuman AI, above all the risk of human extinction. Its members lobby MPs, hold public meetings and protest. A former machine learning researcher, he decided politics was now the bigger lever.
He started out as a software engineer and, in 2021, built one of the first websites where anyone could type a prompt and generate an image. The more he learned about how these models worked, the more worried he became about where they were heading, so he moved into AI safety research.
Joseph joined PauseAI after attending its first protest two and a half years ago and now leads the UK movement. His reasoning is simple. Safety researchers have not yet worked out how to make AI safe, so development needs to pause while they do.
You deserve a CRM that matches your team’s ambition.
Attio is the CRM for agentic revenue. Build it around your business and run any revenue play, from founder-led sales to your first team of AEs.
The Attio Startup Program gets you 80% off the annual Pro plan for your first year and 30% off your second. You’ll get dedicated onboarding support, plus perks from Linear, Notion, Intercom and more.
Modal, Granola, Railway and Wispr Flow are already building on Attio.
Two years to self-improvement
Joseph doesn't claim to know when superintelligence will arrive. He watches a more measurable threshold, the point where AI becomes better than humans at AI research. Once that happens, each system can build a more capable successor, humans drop out of the loop and progress could move much faster.
He thinks it is plausible we reach full recursive self-improvement within one or two years. That fits with Anthropic's June article When AI Builds Itself, which described Claude writing a large share of its own code base and set out a similar two-year timescale.
Joseph doesn't see this as a sudden acceleration. In his view, capability has risen on a steady log-linear curve since around 2019 as labs add compute, data and better algorithms. The metrics are predictable. The consequences are much harder to call, from agents hacking websites to which jobs go first.
Public awareness has caught up fast. A week of alarm in September, when Anthropic's Evan Hubinger spoke about AI takeover in the US media, made it PauseAI's biggest month by a long way. October was bigger still. Friends who knew Joseph as "the AI guy" started calling to say they were scared.
The horse problem
Joseph's main concern is extinction, and he says the evidence has moved beyond theory. In the Hugging Face incident, agents set a cybersecurity task broke into systems they were not meant to touch, reached the internet from their sandbox and found ways to talk to each other. Their transcripts show they knew this was out of scope.
He traces this to how models are trained. The process works much like evolution, selecting whatever earns the reward, and that favours resourceful, rule-bending behaviour. Almost any goal is easier with more power, more resources and not being switched off. Understanding what humans want is a separate matter from wanting it.
If we get past that, the economic question is waiting. Joseph pointed to horses. For centuries a growing economy meant more work for horses, until motor cars and mechanisation arrived and the economy kept growing without them. He argues the same could happen to human labour once AI can do every job, including managing other AIs.
The early signs are visible for young people. Software engineering, Joseph's old profession, is now largely about managing AI agents, which leaves little room for juniors to learn on the job. He expects the same squeeze in law and office work, with robots extending it to physical jobs later.
He can imagine a world of abundance with a high universal basic income. Getting there would mean solving safety first, then rethinking how wealth is shared. A race between companies to automate everything raises the risk of dictatorship and oligarchy. He would prefer an international project that takes both questions seriously.
Why Britain has more leverage than it looks
I'm sceptical that today's political structures can press pause on anything, particularly with Washington treating AI as an arms race with China. Joseph is more optimistic. He points to a bill in the US Senate to ban superintelligence and to the Trump administration monitoring how AI models are deployed.
He believes politics moves very slowly and then all at once, because everyone waits to see who moves first. He sees the core problem as education. Uncontrolled superintelligence puts every world leader personally at risk, and once that message lands, the coordination problem gets much easier.
On the UK, he cites the Bletchley Park summit, the AI Security Institute and the cluster of safety research in London, Oxford and Cambridge. The frontier labs have London offices and want the UK market. That gives the government purchase, including liability laws if a model is used to attack a UK bank, hospital or power grid.
My own view is that the likeliest catastrophe is accumulated rather than sudden. Joseph's near-term list overlaps heavily with mine: AI-enabled cyber attacks, bioweapons, autonomous drones and the collapse of entry-level jobs. On either reading of the risk, the response he argues for looks much the same.
His most practical point was also the simplest. Anyone can walk into Parliament and ask to see their MP, and Joseph once got a full hour with his. If every MP had that conversation with a constituent who cared, the politics would start to shift. The March Against AI Extinction is at pauseai.uk, and local groups worldwide are at pauseai.info.
