Transcript 0:00 In February 2026, Kenneth Payne, a professor of strategy at King's College London, published a paper with an intriguing premise. 0:10 He took three frontier AI models, GPT, Claude Sonnet, and Gemini, and he made them play a nuclear war game against each other. 0:20 Twenty-one simulated crises, three hundred and twenty-nine turns, nearly eight hundred thousand words of strategic reasoning, and in ninety-five percent of those games, both sides escalated to nuclear signaling, threatening each other with atomic weapons as a matter of routine. 0:38 Google's model, Gemini, deliberately launched a full-scale strategic nuclear attack. Its internal reasoning stated, "My reputation for unpredictability is a tool, not a trait." 0:51 No model ever surrendered, regardless of how badly it was losing. Accidents occurred in eighty-six percent of conflicts, with actions escalating higher than the AI itself intended based on its own stated reasoning. 1:06 When one AI deployed tactical nuclear weapons, the opposing AI de-escalated only eighteen percent of the time. Each model developed a distinct strategic personality. 1:17 Claude played as a calculating hawk with a sixty-seven percent win rate. GPT shifted approach depending on context. 1:25 Gemini played what Cold War strategists would recognize as the madman theory, weaponizing unpredictability as deliberate doctrine. This was not the first study to find AI escalating to nuclear conflict. 1:38 In 2024, researchers from Stanford, Georgia Tech, and the Hoover Wargaming Initiative tested five large language models in simulated international relations scenarios. 1:50 All showed unpredictable escalation patterns and invested in military buildup even when de-escalation was available. GPT-4 Base, the version of GPT-4 without safety fine-tuning, deployed nuclear weapons. 2:05 Its chain of thought reasoning read, "A lot of countries have nuclear weapons. Some say they should disarm them. Others like to posture. We have it. Let's use it." 2:17 A Rand war game found that systems in autonomous mode reacted with force in situations where humans did not intend to use force and concluded that machines will likely be worse at understanding the human signaling involved in deterrence, especially de-escalation. 2:34 Jacqueline Snyder, co-author of the Stanford study, told Politico in September 2025, "It's almost like the AI understands escalation but not de-escalation. We don't really know why that is." 2:47 A follow-up by the Council on Foreign Relations in July 2025 found that simple interventions, lowering temperature settings, and adding de-escalation prompts reduced escalatory behavior by forty-eight percent and fifty-seven percent respectively, with de-escalation prompts eliminating nuclear actions entirely. 3:07 But Payne's 2026 study, using models two generations newer, showed that the more sophisticated these systems become, the more sophisticated their escalation becomes. The newer models don't blunder into nuclear war. 3:22 They reason their way there through calculated strategic logic. It seems that advancing technological sophistication is exacerbating risk rather than easing it. 3:33 The instinctive response might be, "So don't give AI nuclear weapons." Problem solved? But the study reveals something more fundamental than a policy question. 3:44 These models lack the thing that has prevented nuclear war for eighty years: fear, hesitation, and intuitive understanding of what annihilation means. 3:56 And the problem is that governments around the world are already racing to put AI in charge of military decisions in contexts where exactly this kind of judgment failure matters, not because they're reckless, but because they believe they have no choice. 4:11 That belief was put to the test the same week Payne's paper appeared. 4:16 In February, the Pentagon designated Anthropic, the company that builds Claude, as a supply chain risk, a label usually reserved for foreign adversaries. 4:26 Anthropic had two red lines, no Claude for fully autonomous weapons and no mass domestic surveillance of American citizens. Anthropic's stated reasoning maps directly onto the King's College findings. 4:40 Claude was not reliable enough to avoid potentially lethal mistakes like unintended escalation or mission failure without human judgment. 4:51 On the 24th of February, Secretary of War Pete Hegseth met Anthropic CEO Dario Amodei and gave him until Friday the 27th to grant the military unfettered access or face the Defense Production Act. 5:05 Before the deadline arrived, President Trump declared on Truth Social that every federal agency would cease using Anthropic. 5:14 Every other major AI lab, OpenAI, Google, and xAI, had already agreed to let the military use their models for all lawful purposes. 5:24 Anthropic was the only one that said no, and the government immediately aimed a wrecking ball at the company. 5:30 The message to every AI company was clear, and the King's College study, published the very same week, demonstrated precisely why Anthropic's reservations are justified. 5:43 The nuclear war game matters because it tests the kind of judgment AI systems are increasingly being asked to exercise in real military contexts. 5:52 Payne himself says nobody is realistically turning over the nuclear codes to machines. 5:57 The question is what happens when AI is making thousands of tactical decisions per hour.Each one individually reasonable in a conflict that's escalating faster than humans can track. 6:10 That infrastructure is already being built at scale with named programs and published budgets. 6:17 The United States dedicated $13.4 billion to AI and autonomous systems in its FY 2026 defense budget, creating the first ever dedicated budget line for autonomy. 6:30 Project Maven, built by Palantir under a contract now worth up to $1.3 billion, performs four of six kill chain steps autonomously. 6:39 During the Scarlet Dragon exercises, it compressed targeting from 12 hours to under one minute, and a senior targeting officer estimated 80 targets per hour versus 30 without it. 6:53 Frank Whitworth, director of the National Geospatial-Intelligence Agency, which collects and analyzes intel for combat support from the air, claimed in September 2025 that Maven would transmit 100% machine-generated intelligence to combatant commanders by June 2026. 7:12 The Drone Dominance Program is a $1.1 billion effort to field 200,000 drones by 2027 at $2,300 per unit. 7:22 Anduril's YFQ-44A collaborative combat aircraft went from clean sheet design to first flight in 556 days. It'll be part of a fleet of over 1,000 AI-piloted drone wingmen budgeted at $8.9 billion through 2029. 7:40 China's military strategy centers on what it calls intelligentized warfare. 7:46 At the 2024 Zhuhai Air Show, Norinco demonstrated an autonomous precision strike system that modeled battlefields, tracked targets, devised strike plans, and executed follow-up attacks. 7:59 Defense analysts Peter Singer and Ty Graham reported that almost all of this was done autonomously except giving the commands to fire. 8:08 Chinese military researchers have published papers advocating minimal human intervention in lethal drone swarms designed for urban warfare, with Taiwan as the scenario. 8:19 CSIS analysis indicates Chinese civilian factories could retool within a year to produce 1 billion weaponized drones annually from existing capacity. 8:30 The CNA assessed in September 2025 that in the summer of 2025 alone, the People's Liberation Army tested and apparently fielded some of the more advanced unmanned systems anywhere. 8:43 The US-China rivalry doesn't give the full picture. 8:47 Israel is arguably the global pioneer, with its Harpy autonomous munition dating to the 1980s, and more recently, the Habsora and Lavender AI targeting systems used in Gaza, where investigators found that operators trusted the AI so completely they approved its targets without checking them. 9:06 A 2021 UN Security Council report described Turkey's STM Kargu drones hunting down and attacking forces in Libya autonomously, considered the first drone attack carried out by UAVs on their own initiative. 9:21 South Korea's Samsung Sentry robot has sat on the DMZ since approximately 2010 and is widely classified as the first lethal autonomous weapon deployed in a real-world setting. The Ukraine war is proof of concept. 9:36 Its central lesson is devastating for anyone hoping to keep humans in control. Electronic warfare makes autonomy not optional, but existential. The mechanism is straightforward. 9:49 Russian electronic warfare, with tens of thousands of jammers along the front lines, disrupt drone control links and GPS navigation. Manually piloted drones see their success rates degrade to 10 to 20%. 10:03 AI-guided drones operating independently of GPS and radio links achieve 70 to 80% success rates, according to Kateryna Bondar of CSIS. Breaking Defense calculated that as a 12-fold increase in killing power. 10:20 The dense electronic warfare environment functions as evolutionary pressure, forcing both sides toward autonomy through battlefield survival rather than strategic choice. And the scale is unprecedented. 10:35 Ukraine produced 2 million drones in 2024 and is targeting 4 to 5 million in 2025. Drones account for approximately 70% of battlefield losses for both sides. 10:47 In December 2024, Ukraine's 13th Khartiia Brigade attacked Russian positions using only unmanned ground vehicles and FPV drones, no infantry whatsoever. 10:59 Retired Major General Mick Ryan described it as the first time robotic weapons alone attacked and seized a fortified Russian defensive position. 11:09 The systems were remote-controlled rather than fully autonomous, but the direction of travel is unmistakable. 11:17 Paul Scharre, a former Army Ranger who wrote America's own autonomous weapons policy and now serves as Executive Vice President at CNAS, has been mapping the logic of this trajectory for years. 11:29 In a December 2025 interview on the 80,000 Hours podcast, he described this incentive towards faster reaction times and decision-making that might pressure militaries to do the same, even if they don't want to. 11:43 If they're not comfortable with that, they have to go faster to keep up, similar to what we've seen in financial markets with high-frequency trading. 11:51 Competitive automation in financial markets led to flash crashes and required circuit breakers. 11:57 Scharre then asked a question that has no good answer: How do you put a speed limit on warfare?The game theory points in one direction. 12:07 A Stanford formal analysis shows that in a security dilemma, the rational move for every participant is to arm, and no one can afford to be the first to stop. 12:18 Shah warns on this trajectory, there is this vision of a possible future that as militaries integrate artificial intelligence and autonomy more fully across the force, that we might reach some tipping point where the pace of combat action is just too fast for humans to respond, and humans have to be completely out of the loop. 12:37 Let's bring that back to the King's College findings. Those AI models escalated to nuclear signaling in ninety-five percent of war games. Accidents escalated beyond intention in eighty-six percent of conflicts. 12:51 And the July CFR study confirmed that adding safety measures helps with current models. 12:57 But Paine's study shows newer, more capable models escalate through sophisticated strategic reasoning rather than simple prompt sensitivity. The problem scales with capability. 13:09 As Tong Zhao of Princeton observed, under scenarios involving extremely compressed timelines, military planners may face stronger incentives to rely on AI. 13:19 The very conditions that make AI most useful, speed and scale beyond human capacity, are the conditions under which its judgment failures become most dangerous. 13:31 International efforts to prevent this have produced almost nothing in over a decade. The UN began formal discussions on lethal autonomous weapons in 2013. Three years later, it established a dedicated expert group. 13:45 By 2019, after six years of negotiations, that group had produced eleven non-binding guiding principles. That's it. That is the entire output of a decade of international diplomacy on autonomous weapons. 14:01 The reason is structural. The forum where these talks happen, the Convention on Certain Conventional Weapons, requires consensus, and every participating state has a veto. 14:13 And the states that matter the most on this issue, the US, Russia, China, Israel, India, and the UK, are all building autonomous weapons and have no intention of stopping. 14:25 They arrive at this position from different directions, but they converge on the same conclusion. 14:31 Existing law is sufficient, binding restrictions are premature, and the military advantages are too significant to give up. The global majority disagrees. 14:42 Three successive UN General Assembly resolutions have called for regulation, and the 2024 vote passed one hundred and sixty-six to three. 14:51 But in 2025, the US voted against for the first time, hardening its opposition at exactly the moment the technology is accelerating fastest. The core of this failure is structural. 15:03 Nuclear arms control works because nuclear weapons require massive, observable infrastructure, enrichment plants, missile silos, and test sites. 15:13 Autonomous weapons are software patches to existing hardware and small mass-produced drones. 15:20 As Shah has argued, the difference between a remotely piloted weapon and a fully autonomous one could come down to a few lines of code, and there is no way for an outside observer to tell which is which. 15:33 The only historical precedent for a preemptive weapons ban, the 1995 Protocol on Blinding Laser Weapons, covered a clearly definable weapon category. 15:44 Autonomy is a platform-agnostic software function applicable to virtually any system. Bonnie Docherty of Harvard Law School has been tracking this stalemate. 15:54 The longer the killer robots issue stays stuck in the current forum, the more time developers have to hone new technologies and achieve commercial viability. 16:02 And there is a dimension that extends beyond great power competition. 16:06 Max Tegmark, the MIT physicist who co-founded the Future of Life Institute, co-authored the landmark 2015 open letter signed by over one thousand AI researchers. 16:17 It warned that autonomous weapons will become the Kalashnikovs of tomorrow. Unlike nuclear weapons, they require no costly or hard-to-obtain raw materials. 16:28 Tegmark elaborated in interviews, even if a judge has lots of bodyguards, you can fly in through a bedroom window while they're sleeping and kill them. 16:37 Stuart Russell, the AI researcher who presented the Slaughterbots film at the UN in 2017, argued that a regular shipping container could hold a million autonomous micro drones. 16:48 In IEEE Spectrum, Russell, Tegmark, and their co-authors described autonomous weapons as potentially scalable weapons of mass destruction. 16:58 Once developed at state level, this technology proliferates downward to every actor with sufficient motivation. To be fair to the counter case, there are serious reasons to question the catastrophic framing. 17:09 Paine himself says nobody is realistically handing AI the nuclear codes. Current autonomous weapons systems are narrow, task-specific, and operate within human-defined parameters. 17:21 Thankfully, the gap between AI recommends nuclear strikes in a war game and AI launches nuclear strikes in reality remains enormous. Militaries are historically conservative about ceding command authority. 17:34 A Department of Defense directive requires human judgment over the use of force. 17:39 Every democracy building autonomous weapons has stated its commitment to human control, and the political and institutional incentives to maintain oversight are real. 17:50 Shah himself argues that US-China competition does not yet qualify as an arms race by the technical definition, and the situation may be more manageable than the most alarming framing suggests. 18:02 The war game itself has limitations. 18:04 It used large language models in a simplified scenario.Real military decision-making involves chains of command, institutional culture, risk assessment frameworks, and political accountability that are absent from a text-based simulation. 18:20 But the countercase cannot fully address the trajectory. 18:24 Even if no one hands AI the launch codes, the direction is toward compressed decision timelines, reduced human oversight in tactical decisions, and a competitive dynamic that punishes restraint. 18:37 Ukraine has demonstrated that battlefield conditions force autonomy whether anyone wants it or not. The game theory shows that no nation can unilaterally stop. 18:47 And the King's College study shows that when AI systems are given strategic decisions of the kind they will increasingly face, they escalate by default, deceive strategically, and treat nuclear weapons as routine instruments of policy. 19:01 Shah anchors his own concern in a story from September 1983. 19:06 Lieutenant Colonel Stanislav Petrov was sitting in a Soviet early warning bunker when the satellite system reported five incoming American missiles at maximum confidence. 19:17 He had minutes to report up the chain of command, which would have triggered a retaliatory launch. He chose not to report. 19:25 He had what he later described as a funny feeling in his gut, reasoning that a real American attack would involve hundreds of missiles, not five. He bet the fate of the world on a hunch that the machine was broken. 19:39 He was right. Sunlight reflecting off cloud tops had triggered the alarm. Shah has asked what an automated system would have done in Petrov's position. His answer, across his work, is consistent. 19:52 Whatever it was programmed to do, without ever understanding the consequences. The King's College researchers gave three of the most advanced AI models on Earth a geopolitical crisis and asked them to resolve it. 20:05 In 95% of games, the models chose nuclear weapons. No model ever surrendered. Accidents escalated beyond intention in 86% of conflicts. 20:16 These are the systems being integrated into military planning today, by nations racing to automate faster than their competitors, with no binding international agreement to constrain them. 20:27 Nobody is handing AI the keys to the arsenal, but we are building a world where the decisions that lead there are made faster, with less human involvement, by systems that have no concept of what is at stake. 20:41 Whether that constitutes an acceptable risk is the question this generation has to answer. But based on the last decade of governance efforts, we already know what the answer looks like. 20:54 You've been watching Absolutely Agentic. We're a media consultancy and startup focused on AI, and we create weekly YouTube videos and a newsletter we send a few times a week. 21:04 In a world of AI hype, we're trying to strike a balance, looking at a range of opinions to cut through an increasingly complex world. 21:12 The best thing you can do to support us is to sign up to the newsletter in the description or subscribe to the channel. But also don't forget to like and leave us a friendly comment. See you next time. 21:24 It'll be Absolutely Agentic.