This website uses cookies

Read our Privacy policy and Terms of use for more information.

In partnership with

Most of the conversation about catastrophic AI risk assumes the machine is the problem. It develops goals of its own, resists being switched off or slips out of our control. Those scenarios deserve attention, and they share one assumption: that something goes wrong inside the system.

There is another version in which nothing goes wrong inside the system at all. The model simply does what it is asked. The danger lies in what humans choose to do with it.

In September, Bill Gates said AI is "certainly powerful enough to drive events that, you know, cause a billion deaths". He added that no weapon has ever matched the combination of people with ill intent and the latest AI tools.

That marks a shift. Gates spent two decades funding global health and wrote a book on pandemic preparedness, yet in 2023 he described AI risks as real but manageable. He was also clear about the ranking, putting human misuse ahead of a rogue system.

Leading labs appear to agree. In May 2025 Anthropic released Claude Opus 4 under its ASL-3 standard, with filters on weapons-related questions, because it could not rule out meaningful uplift. A month later OpenAI said its next models would likely reach high capability in biology.

Want to get the most out of ChatGPT?

ChatGPT is a superpower if you know how to use it correctly.

Discover how HubSpot's guide to AI can elevate both your productivity and creativity to get more things done.

Learn to automate tasks, enhance decision-making, and foster innovation with the power of AI.

The barrier that has held

Historically, three things stood between wanting a biological weapon and having one. Information has been eroding for decades through open publication. Access to materials still depends on DNA synthesis screening, though membership of the main screening consortium is voluntary.

The Biden executive order of October 2023 attached screening requirements to federally funded US research. It was revoked in January 2025, leaving the one physical safeguard more fragile than it was.

The barrier that has actually stopped people is tacit knowledge, the practical skill that lives in a practitioner's hands. Aum Shinrikyo had around a billion dollars, trained scientists and its own facilities. Its 1993 anthrax release harmed nobody because it had used a harmless vaccine strain without realising.

Islamic State held territory, raised hundreds of millions and recruited graduates. A laptop recovered in 2014 showed ambitions in biology, yet the group never got there. Across the record, the barrier that held was competence, and competence is what general purpose AI is built to supply.

No off switch for open models

Frontier models from the big labs mostly refuse dangerous requests, and their makers can patch them overnight. Open weight models from Meta, DeepSeek, Mistral, Alibaba and Moonshot work differently. Once downloaded they cannot be recalled, and safety training can be stripped out for a few thousand dollars.

In July, the British security firm Mindgard jailbroke two Moonshot models, and the outputs included a plan for a biological attack. The jailbreak persisted in the app's memory across sessions and server restarts. Moonshot opened an internal review in October, after the BBC got in touch.

Mindgard's researchers say American models share the same weakness, so the issue extends well beyond one company. The real concern is permanence. A safeguard that fails on a downloadable model fails for every copy already in circulation.

There is a fair counter-argument. A 2024 RAND red team exercise found no statistically significant uplift from language models when teams planned an attack. Reading a procedure is a long way from carrying it out, and nobody has shown the Moonshot outputs would work.

RAND, however, tested early 2024 models. Anthropic and OpenAI assessed systems two years newer, with stronger reasoning and tool use, and reached more cautious conclusions. Every open release is permanent, so any measurement is a snapshot of a moving target.

A weapon with no return address

Biological weapons have not shifted the balance of power because building them needed something close to a state programme, with facilities, budgets and specialists. That kept the capability inside a small club that treaties could monitor. If AI compresses the requirement, the club stops being a club.

A sanctioned state's nuclear programme can be photographed from orbit and struck. A capability developed largely in software offers no centrifuge hall to target. The pattern echoes recent warfare, where drones costing a few thousand dollars destroy tanks worth millions.

Deterrence depends on knowing who attacked. A pathogen emerges in a city, and by the time anyone establishes whether it was engineered or natural, it may have spread to dozens of countries. Modern infrastructure adds to the risk, because utilities and food supply run with little slack and depend on staff turning up.

Responses fall into three camps. Gates wants government mandates, arguing that "no one thinks self-regulation is enough". A second camp wants universal, mandatory DNA synthesis screening. A third accepts that open weights cannot be recalled and invests in faster vaccines, wastewater detection and stockpiles.

The shift worth watching is the arrival of a weapon that small actors may soon be able to hold and large powers cannot retaliate against. How we govern that gap will shape global security for a generation.

Watch last week’s video

Reply

Avatar

or to participate