Transcript 0:00 Nobel Prize-winning computer scientist Geoffrey Hinton, known as the godfather of AI, quit his job at Google so he could be more open about AI risks Does humanity know what it's doing? No. 0:14 He thinks there is a twenty percent chance that AI will wipe out humanity. He's not alone. Nearly half of AI researchers think there's at least a one in ten chance that their work could end in human extinction. 0:31 It's not a fringe view or media hype. Two thousand seven hundred and seventy-eight researchers were surveyed in the largest study of its kind. 0:39 The Hollywood view of AI takeover is dominated by images of killer robots and prolonged wars with machines. 0:47 The predictions of experts are more chilling, like the sudden wipe out of humanity in an afternoon after a super intelligent AI coldly decides we're standing in the way of its goals. 0:59 But there's another possibility, one I find much more plausible. Let's imagine a not so distant scenario [tense music]. 1:13 It's the winter of twenty twenty-nine, just over a year since China invaded Taiwan. 1:20 America and China are now locked in hybrid warfare across the Pacific while seeking a decisive strategic advantage by racing towards super intelligent AI. 1:33 In Europe, an uneasy ceasefire holds in Ukraine, but the continent is reeling from hacks on its power grids, airports and infrastructure. The NASDAQ has collapsed. 1:46 TSMC, the world's biggest chip manufacturer, is now under Chinese control. Unemployment is climbing past ten percent across the West. 1:58 You haven't lost your job yet, but more than half of your department have been replaced by AI agents. The news is chaos. When you scroll down a feed, it's impossible to tell what is real. 2:11 Independent media is a shell of its former self, its demise sped up through the use of chatbots as sources of information. Outside, there are cameras everywhere. 2:23 Robot dogs patrol the street while delivery drones whiz above them. You're just not sure which ones are watching you. This isn't a sudden AI apocalypse. 2:34 There are no killer robots or super intelligence seizing control, just everything seems to be getting progressively worse, less human, more machine-like and dystopian. But this 2:50 is just the beginning [tense music] [static]. The opening scenario matches what's been called the cumulative AI x-risk hypothesis. 3:07 The term comes from a twenty twenty-four paper by Atossa Cassazade. 3:12 Her argument is that we've been too focused on the science fiction scenario, a super intelligence suddenly seizing control, while ignoring a more realistic pathway, 3:23 one where smaller disruptions accumulate and converge, each individually manageable but together eroding economic and political structures until a triggering event causes irreversible collapse. 3:38 The sum of their parts becomes exponentially greater than the whole. The artificial intelligence revolution is often compared to the Industrial Revolution. 3:48 It's a huge technological shift that transforms, well, basically everything. Life today would be largely incomprehensible to a person from the early eighteenth century. Most people live short hard lives working the land. 4:04 Over the next two hundred years, industrializing countries urbanized, populations grew rapidly, and infant mortality dropped dramatically. 4:13 AI could well have similar transformative benefits and probably much, much faster. But the Industrial Revolution also carried a very subtle but catastrophic accumulative risk: climate change. 4:29 Initially, there simply wasn't the ecological knowledge or really enough countries burning fossil fuels to know that this would be an unintended consequence. Yet no single weather event is likely to end civilization. 4:42 Instead, greenhouse gases accumulate in the atmosphere, which traps heat and creates a series of adverse but cascading effects. 4:51 Ice sheets melt, which disrupt ocean currents, which changes precipitation patterns, which causes crop failures, which drives migration, which destabilizes governments. 5:02 All the while, retreating ice sheets reflect less sunlight, and the feedback loop accelerates. 5:10 In his book The Coming Wave, Microsoft AI CEO and DeepMind co-founder Mustafa Suleyman captures the core dilemma: sooner or later, a powerful generation of technology leads humanity toward either catastrophic or dystopian outcomes. 5:28 One of the clearest benefits of AI is democratizing access to skills and knowledge. So long as they have an internet connection, even the least educated people in the world can now access large language models. 5:39 It won't be long, if they haven't already, until these machines surpass human capability in coding, scientific research and even running businesses. 5:49 This could represent a step change in productivity and discovery before we even need to get to super intelligence. But trade-offs come in multiple forms. 5:58 The first is bad actors gaining access, terrorists or hackers obtaining self-replicating viruses or building sophisticated weapons like micro dronesBut there's also a more current example. 6:12 Before 2022, producing imagery was hard. Illustration took years to learn. I did art at school, and my best paintings took over 20 hours each. 6:24 But then Midjourney and other AI image generators have come along and enabled basically anyone to generate digital art and simulated photography at a fraction of the previous time and cost. 6:37 Photographers, designers, and stock libraries are suddenly displaced. Whether any of this is even legal remains unclear. If enough people use it, the law can't keep up. This is one of the core themes of The Coming Wave. 6:53 AI brings benefits, and most people won't forego those benefits unless the consequences affect them directly, so the risks are unlikely to be contained. Accidents will happen, and bad actors will exploit them. 7:07 Solomon calls these fragility amplifiers, less discrete events, less a robot attack or deepfake video, and more a slow and diffuse process undermining foundations. And crucially, we might not notice until it's too late. 7:23 Each individual problem looks manageable. We tell ourselves we've got it under control, but we're not paying attention to the cumulative effect or how much resilience we're burning through with each fix. 7:37 Referring back to the survey of nearly 3,000 AI researchers, when asked what concerned them most, it wasn't extinction. 7:45 Over 70% expressed substantial or extreme concern about AI spreading misinformation and manipulating public opinion. 7:54 Similar levels of concern were expressed about authoritarian rulers using AI to control their populations. 8:00 Because it seemed more plausible, the slow erosion of truth and freedom worried them more than superintelligence wiping us out. In her paper, Cass Zadeh organizes these concerns into six overlapping categories 8:16 under the acronym MISTER. This stands for manipulation, insecurity, surveillance, trust erosion, economic destabilization, and rights infringement. 8:29 She calls it a perfect storm situation, where not only is recovery extremely challenging, but the potential for irreversible collapse is a stark reality, and the frightening thing is you don't have to look hard to see them already. 8:46 Manipulation and trust erosion go hand-in-hand, so let's start there. Falling trust in institutions and the media has correlated closely with the rise of social media and smartphones. 9:00 A Gallup poll shows that trust in the media has sunk to an all-time low. There are many reasons for this: the Iraq War, the financial crisis, COVID-19, double-digit inflation. 9:11 But technology acts as a kind of rocket fuel. We're already spending nearly four hours a day staring at smartphones, and total daily screen time across all devices is around 7.5 hours. 9:25 Research from Harvard Business Review suggests AI will only accelerate that dependence. In 2016, the year of Brexit and Trump's first election win, post-truth became the Oxford Dictionary Word of the Year. 9:41 Fake news was Collins' Word of the Year. By 2024, there was great concern that AI video generators would muddy the next election, but people were wary, and the technology wasn't quite there yet. 9:55 A year later, video and image generators have near parity with real photography, and they're only getting better. 10:02 While social networks try to label AI-generated content, it's easy to generate and share a deepfake that slips through. 10:11 Female politicians, already targets of the most violent forms of social media hatred, find themselves in the firing line. 10:19 The Italian Prime Minister Giorgia Meloni had her face swapped into a porn scene, a video viewed millions of times. It took nearly half a decade for the perpetrators to face consequences. 10:31 Hong Kong pro-democracy campaigner Carmen Lau discovered that AI images portraying her as a sex worker had been sent to her neighbors. An actual person wouldn't even need to be involved. 10:43 You could set up an AI agent to prompt the latest video models ad infinitum. This matches Steve Bannon's infamous approach: flood the zone with shit. 10:56 In Cass Zadeh's words, "The information ecosystem becomes so polluted that rational public discourse becomes nearly impossible." 11:05 Perhaps we'll become so mistrustful of digital media that social media eats itself, or that AI eats it alive, the subject of my last video. Content by bots for bots, essentially dead internet theory. 11:22 My Instagram feed is already a carousel of AI slop. My LinkedIn inbox is full of AI-generated sales messages, so many that I've stopped opening any of them. This is being funded at the highest levels. 11:36 Andreessen Horowitz backed a startup called Doublespeed, whose pitch reads, "We've built an attention intelligence platform to build and deploy social agents at scale across every demographic and niche. 11:49 Our autonomous creators learn from every interaction and become more valuable over time, driving conversions and engaging with millions 24/7." 11:59 So a major Silicon Valley firm is funding an AI phone farm that breaks the terms and conditions of the platforms where it operates. 12:07 Meanwhile, Meta has cut its moderation teams in favor of a Community Notes-style system.This alternative reality is already playing out in real time. 12:18 A recent analysis found that social media posts stating London is dangerous have spiked dramatically from eight hundred and seventy-four posts in 2008 to over two hundred and fifty-eight thousand in 2024. 12:31 Many come from new accounts with AI-generated profile pictures that exclusively post this content. The reality, London's homicide rate hit its lowest level in years. Violent crime fell in all 32 boroughs. 12:48 But the narrative keeps spreading anyway, shaping how people feel about the city, whether tourists visit, and even how residents vote. Who's behind it? It's difficult to know, but we might well have our suspicions. 13:05 Trust falls further because people simply don't know what to believe. Lawyers and even judges have cited previous cases in court that turned out to be AI hallucinations, cases that never existed. 13:18 A similar pattern played out in November 2025 when West Midlands Police banned Israeli football fans from attending a match at Villa Park. 13:27 The intelligence report justifying the ban included details of a previous match, a match that never happened. It was generated by AI. 13:36 Senior police leaders twice denied to MPs that AI had been used, but it had, and by January 2026, it had cost the chief constable his job. This wasn't a chatbot getting a quiz question wrong. 13:51 It was an AI hallucination shaping police decisions, leading to discrimination. This leads into the theme of insecurity. 14:01 The Netflix movie Leave the World Behind sees a family retreat to the country, only for America to come under massive infrastructural attack by an unseen and unnamed enemy. 14:12 It feels like the accumulative risk hypothesis playing out on screen, but this kind of hybrid warfare isn't fiction anymore. 14:20 In The Coming Wave, Suleyman argues that advanced technology will fragment power away from nation states. 14:27 On the positive side, this could enable extraordinary crowdsourcing, medical breakthroughs from ordinary people working with AI assistants. But the same democratization can work in reverse. 14:40 A small private lab could use AI to engineer biological weapons. In 2022, researchers at Collaborations Pharmaceuticals demonstrated how easily this could happen. They reversed the logic of their drug discovery AI. 14:54 Instead of avoiding toxic molecules, they asked it to generate them. 14:58 Within six hours, the model had designed forty thousand potential chemical weapons, including VX nerve agent, despite never being trained on warfare agents. 15:09 Hackers, whether state-sponsored or working alone, have been emboldened. Britain has experienced something of a hacking epidemic. Marks & Spencer, Harrods, and Heathrow Airport have all been hit in recent years. 15:23 In autumn 2025, Jaguar Land Rover went offline for weeks after a ransomware attack, which had major repercussions through UK car manufacturing supply chains. 15:35 In September 2025, Anthropic detected what they described as the first large-scale cyber attack executed without substantial human intervention. 15:44 A Chinese state-sponsored group had manipulated Anthropic's Claude code tool to autonomously infiltrate roughly 30 targets: technology companies, financial institutions, chemical manufacturers, and government agencies. 15:58 The AI performed eighty to ninety percent of the campaign, making thousands of requests, often multiple per second, an attack speed human hackers simply couldn't match. 16:10 This is a new form of hybrid warfare aimed at disruption and instability rather than physical destruction. 16:17 As we become ever more connected through the Internet of Things, we become increasingly vulnerable, and hacking is only one front. 16:24 Also in September 2025, European airspace was invaded by mysterious drone swarms, closing airports and scrambling Danish and Polish fighter jets. 16:34 Without better safeguards, physical sabotage will become more common as bad actors gain access to cheaper airborne technology. 16:42 Drones will also get smaller and potentially more deadly, making them more difficult to contain and know who the perpetrators are. 16:50 Slaughterbots, a viral short film released by the Future of Life Institute in 2017, shows the potential in chilling and very well-aged detail. This is how it works. 17:04 [drill whirring] [gunshot] [audience applauding] Did you see that? That little bang is enough to penetrate the skull 17:21 and destroy the contents. All of this has the effect of destabilizing democratic governments who appear slow to react and clueless to identify the real culprits. 17:31 The public loses faith in institutions that seem powerless against threats they can barely comprehend. 17:38 The ability for bad actors to project power for adverse ends leads to the possibility of increased surveillance and rights infringement. 17:47 We only really have to look back at the COVID-19 pandemic to see how quickly basic freedoms can be curtailed and accepted as a new normal in the face of an unprecedented danger. 17:57 People largely accepted government pleas to stay at home, and some even reported their neighbors for infringements. What would it take for such measures to come about at the hands of AI? 18:08 Massive cyber attacks or bioterrorism? 18:11 Or perhaps an AI lab getting close to developing super intelligence, triggering intense surveillance that then spreads throughout societyThis sort of action is enabled by much better facial recognition, location tracking, and AI's ability to handle massive data sets. 18:29 We can already see this through the deployment of ICE across American cities. 18:33 The US government's Immigration Enforcement Agency has signed a $30 million deal with AI data analytics company, Palantir, to build a surveillance platform. 18:44 According to The Telegraph, a conservative-leaning newspaper and website in the UK, the agency is also using facial recognition technology, spyware, mobile phone location databases, drones, and un-uniformed contractors to track where people work, eat, and sleep. 19:04 The US government is already attempting to bend private AI companies to its will, with chilling effects. 19:11 In February 2026, the Pentagon gave Anthropic an ultimatum to remove all restrictions on how its Claude AI model could be used by the military, or lose its $200 million classified systems contract. 19:25 Anthropic's two red lines were really quite narrow: no mass surveillance of US citizens, and no fully autonomous weapons without a human in the loop. 19:36 The Pentagon said it had no intention of doing either of those things, but when Anthropic asked for that commitment in writing, the contract language would have allowed those safeguards to be overridden at will. 19:48 When Anthropic refused, President Trump ordered every federal agency to cease using Anthropic's technology, 19:56 and the Secretary of War designated the company a supply chain risk, a label normally reserved for foreign adversaries. 20:05 If the state genuinely didn't want these capabilities, it wouldn't need to crush the company refusing to provide them. As we've seen, the surveillance infrastructure to make it possible is already being built, 20:17 and Anthropic CEO Dario Amodei laid out the potential risks in an interview with CBS. One is domestic mass surveillance. 20:27 They're worried that, you know, uh, things may become possible with AI that weren't possible before. 20:35 An example of this is something like taking, um, uh, uh, um, data collected by private firms, having it bought by the government, and analyzing it in mass via AI. That actually isn't illegal. 20:47 Um, it was just never useful before the era of AI. So there's this way in which domestic mass surveillance is get- is getting ahead of the law. 20:55 The, the, the technology is advancing so fast that it's out of step with the law. We're already carrying tracking devices everywhere we go. 21:04 But in 2024, Harvard students demonstrated something more unsettling: AI-powered glasses that could identify strangers and pull up their home address within seconds just by looking at them. 21:15 They achieved this through hooking Meta's Ray-Ban smart glasses up to a facial recognition system. 21:21 Dutch journalist and tech expert Alexander KlΓΆpping demonstrated such glasses on the streets in Amsterdam, instantly pulling up personal information, including names, professions, and social media profiles. 21:35 Founder of the Future of Life Institute, Max Tegmark, writes a harrowing passage in his book Life 3.0, imagining how a superintelligent AI could slowly undermine the world order through media manipulation and market control. 21:50 He describes how the step from the perfect surveillance state to the perfect police state would be minute. 21:57 A world where everyone is required to wear a security bracelet that uploads your position, health status, and overheard conversations. Try to remove it, and it injects a lethal toxin. 22:09 [dramatic music] If one nation gained a decisive strategic advantage through discovering superintelligence first, they could subjugate the rest of the world, then deploy such technology to become an immovable totalitarian regime. 22:25 This is something philosopher Nick Bostrom has named the Singleton, a world order that has permanently ended all threats to its supremacy. 22:34 [dramatic music] Returning to the reality of the present, you don't have to look hard to see the MRST acronym in action already. 22:43 Increased surveillance and rights infringement are happening right now, and could well become more pervasive if perceptions of instability also rise. 22:53 Large-scale cyber attacks seem to be getting more common, and drone technology has increased dramatically, having been deployed widely during the Ukraine war. 23:04 Economic destabilization is probably the most uncertain of Cass Zadeh's categories. 23:09 AI's impact on employment is already affecting some industries, but it's not widespread yet, and there could be job creation as well as displacement. 23:19 But there are widely held fears that it could sigh through the job market, and predictions from those building the technology are stark. 23:27 In May 2025, Dario Amodei warned that AI could wipe out half of all entry-level white-collar jobs and spike unemployment to 10 to 20% in the next one to five years. 23:39 AI companies and government need to stop sugarcoating what's coming, the possible mass elimination of jobs across technology, finance, law, consulting, and other white-collar professions. 23:51 Amodei also explored many of the themes raised so far in a recent essay titled The Adolescence of Technology, which we've covered in an earlier video. 24:01 The shift from AI as a tool to AI as a replacement would happen faster than most people expect. "It's going to happen in a small amount of time, as little as a couple of years or less," Amodei has said. 24:15 LinkedIn's chief opportunity officer, Aneesh Raman, wrote in The New York Times that AI is already breaking the bottom rungs of the career ladder, with junior developers, paralegals, and analysts at highest risk.I've explored the jobs conundrum in some detail in an earlier video, but there is a possible future where AI generates such incredible wealth that a few years of rising unemployment becomes a footnote. 24:40 Perhaps a new AI-based economy creates jobs that don't exist yet. Amodei himself acknowledges that technology could cure diseases and drive extraordinary growth, but he also painted a bleaker picture. 24:53 Cancer is cured, the economy grows at 10% a year, the budget is balanced, and 20% of people don't have jobs. 25:01 If unemployment did reach 20%, we'd be approaching territory not seen since the Great Depression, when the rate peaked at 25% in 1933. 25:10 That era brought breadlines, bank failures, and the rise of political extremism across Europe and America. The old adage holds, any society is only three meals away from revolution. 25:26 If liberal democratic institutions are swept away in such a crisis, history suggests something else fills the vacuum. 25:33 Military-industrial coalitions, tech oligarchies, or AI-powered surveillance states could rise from the ashes. The singleton becomes all the more likely. 25:45 So we have all six letters of Cassisadi's MISTER framework already visible. Manipulation, insecurity, surveillance, trust erosion, economic destabilization, and rights infringement. Each one individually manageable. 26:02 Each one being managed, or so we tell ourselves. But what happens when they compound? Let's return to that scenario I outlined at the beginning. 26:13 China's seizure of Taiwan, and with that, some 70% of the world's semiconductor production, acts as a catalyst for Cassisadi's accumulative risk hypothesis to take shape from 2030. The world is in a new Cold War, 26:30 but even more tense than the early 1960s. The semiconductor shortage is biting hard. With TSMC under Chinese control, chip prices triple within months. 26:45 Automobile plants in Michigan halt production lines. Apple delays the iPhone 22 indefinitely. Data centers cannot expand capacity at the very moment AI compute demand peaks. 27:02 Naval skirmishes between China and the US in the Philippines Sea become more frequent, stretching to a drone battle and naval standoff a few nautical miles from Guam. 27:15 1960s-style duck-and-cover nuclear drills return to schools across America. NVIDIA's stock, which topped $312 in 2027, falls below $40 by March 2030, a nearly 90% loss from its peak. 27:33 The contagion spreads through pension funds and retail portfolios, heavily weighted toward technology during the boom years. Prominent AI startups collapse in quick succession as venture capital dries up. 27:47 Stability AI files for bankruptcy in April. By the summer, even OpenAI is forced into emergency restructuring after Microsoft writes down its investment. 27:58 Yet AI development continues, funded by a new military-led complex, with each superpower racing for a decisive strategic advantage in hybrid warfare and drone technology. 28:11 Palantir soon becomes the world's most valuable company, while export controls and counter-sanctions fragment the global technology supply chain into hostile blocs. 28:22 US unemployment reaches 8.2%, more than doubling from two years earlier. Former manufacturing towns and cities in the Midwest see rates nearing 20%. 28:34 Tax receipts dry up and welfare systems buckle under structural worklessness. Military spending, meanwhile, continues to rise, pushing American state debt past 250% of GDP. 28:49 The information environment predictably degrades throughout 2030 and 2031. 28:55 A week before the German federal election, a fabricated video of the Christian Democrat candidate accepting a briefcase from a Russian oligarch circulates, verified on a blockchain, before a small team of citizen journalists are able to prove it's a fraud. 29:10 By then, the video has more than 200 million views on social media and has been run on what's left of the broadcast media. 90% of German voters saw it. 29:21 The candidate loses narrowly, but the rest of Europe quickly moves on, more than a little distracted by the potential of nuclear missiles firing across the Pacific. Cyberattacks on critical infrastructure intensify. 29:36 Techniques pioneered by Chinese state-sponsored groups during the Taiwan conflict are passed on to criminal organizations and smaller hostile actors. 29:45 In March 2031, a coordinated attack on the Western interconnection grid, later attributed to China's Ministry of State Security, causes rolling blackouts across California, Oregon, and Washington for 11 days. 30:01 Insurance companies begin refusing to underwrite cyber risk for critical infrastructure. China also faces its own internal ransomware attacks, costing billions of dollars. By 2032, US unemployment reaches 14.3%. 30:19 Food bank usage has tripled since 2028. The surveillance apparatus expands, justified by domestic unrest and foreign infiltration. 30:30 Police departments in major cities deploy autonomous patrol drones equipped with facial recognition.In November 2032, a drone in Atlanta identifies a Black man as matching a suspect profile of a food bank looter and pursues him into a parking garage, where a police patrol robot shoots and kills him. 30:50 He had nothing to be afraid of other than to be pursued by a drone. Protests erupt in 30 cities within 48 hours. The National Guard is deployed in 12 states. 31:03 The triggering event comes during an unusually cold February 2033. 31:08 While extreme climate change is dismissed by AI-generated pundits on TikTok, a cascading failure in financial clearing systems freezes electronic payments across North America and Western Europe. 31:20 The technical cause is later attributed to legacy code vulnerabilities exploited by an AI-assisted attack. For three days, credit cards don't work. ATMs dispense no cash. 31:32 Supermarket shelves empty as supply chains seize up. But that freeze is only the beginning. When systems come back online, nothing adds up. 31:43 The attack has corrupted the underlying records, and backup systems have copied the corruption perfectly. A retiree in Ohio logs into her account and finds 40 years of savings reduced to zero. 31:58 A family in Texas discovers their paid-off home now shows an outstanding mortgage. These are not glitches. The correct information no longer exists anywhere. 32:09 Banks try and reconstruct the records by hand, but there is no paper trail to follow. Several major banks collapse within weeks. Businesses cannot make payroll. 32:19 Unemployment spikes well north of COVID levels, but there is no furlough or safety net. Throughout 2033, unrest spreads. Food riots break out across the American South and Midwest. 32:32 General strikes paralyze France, Italy, and Spain. Military units deployed to suppress protests refuse orders or defect. The political vacuum fills rapidly. 32:43 In the United States, an oligarchy of technology executives and military officers announces an emergency administration in late 2033, citing the twin failures to restore basic services and protect national security against Chinese aggression. 33:00 Yet no one can confirm if the enemy is real, as China faces its own societal breakdown worse than the Cultural Revolution 60 years before. The patterns emerge across Europe. In France, another republic collapses. 33:16 Germany fragments as Bavaria and several western states reject Berlin's authority. The United Kingdom fractures into separate English, Scottish, and Welsh states, with sectarian violence returning to Northern Ireland. 33:34 Italy falls to a post-communist coalition that evolves into a new Red Terror. The new regimes share common features. They rely on AI-powered surveillance systems expanded during the preceding crisis. 33:48 Facial recognition, behavioral monitoring, and predictive policing provide the means to neutralize opposition before it organizes. Political opposition is driven underground. 33:59 Independent media is brought under state supervision. The external threat from China justifies internal repression. 34:07 But in this breakdown of statehood, psychopaths are much closer to dangerous levers, and they use AI to get even nearer. 34:15 Old state-controlled systems fall into the wrong hands, until forces are unleashed that no single power can possibly control. 34:26 In February 2034, a hemorrhagic fever outbreak begins simultaneously in Frankfurt, Chicago, and Madrid. It kills 3% of those infected, and recovering from it rarely offers immunity. 34:40 The synchronized release points to deliberate action, but no group claims responsibility, and attribution proves impossible. Western governments, already hollowed out, cannot mount an effective response. 34:53 The surveillance apparatus that justify the new regimes can't contain it. The virus spreads faster than the systems can track, and people won't stop dying. Healthcare workers abandon their posts. 35:06 Supply chains that were barely functioning seize up entirely. By summer 2034, major cities are losing power for days at a time. By autumn, they're losing it for weeks into months. 35:18 Survivors flee urban areas for settlements in the countryside. The authoritarian order doesn't fall to revolution. There is little left to govern. Police drones still patrol empty streets. 35:31 Surveillance cameras still record. There's no one left to watch the footage, no functioning state to act on it. The surveillance infrastructure outlasts the society it was built to control. 35:49 The scenario is speculative, but the mechanisms that could produce it are already in motion. The question is whether containment is possible. 35:59 Cassazade argues that decisive and accumulative AI risks require different responses. 36:05 The decisive risks, the superintelligent scenarios, call for centralized control, like the kind of frameworks that were built for nuclear weapons. 36:16 Accumulative risks need something else: distributed monitoring systems that track how multiple AI impacts compound across different domains, networks of oversight bodies watching specific sectors and sharing data about emerging risk patterns. 36:31 How do you measure when trust is eroded enough to matter? When does economic fragility tip into crisis?Tracking a single catastrophic event is straightforward. 36:41 Tracking the slow compounding of a hundred smaller disruptions is a problem we haven't solved. In The Coming Wave, Mustafa Solomon reached similar conclusions. 36:52 His vision of containment was not a single policy or technology, but what he described as a set of interlinked and mutually reinforcing mechanisms for maintaining societal control of technology during a time of exponential change. 37:06 He outlined ten components, spanning technical safety and audits through to capable governments, international alliances and public movements, with coherence ensuring all elements work in harmony. 37:19 Both frameworks share a common insight. Containment is not a destination. It's an ongoing process. Solomon imagined containment as a narrow path, wreathed in fog, with a precipice on either side. 37:33 On one side, total openness to all experimentation leads to catastrophe. On the other, total surveillance and closure leads to dystopia. The only way through is to maintain balance, step by step, forever. 37:49 In the scenario I described, that balance was not maintained. The triggers were real enough. Geopolitical conflict, economic fragility, technological disruption, declining trust. Each crisis was manageable in isolation. 38:03 Together, they overwhelmed the capacity to respond. 38:06 The accumulative risk played out across five years, each disruption weakening the resilience available for the next, until the system collapse pushed civilization past the point of recovery. 38:18 At the time of writing, some efforts at containment are underway. National AI safety institutes have launched. 38:24 AI companies have published responsible scaling policies, introducing safety levels modeled on the biosafety frameworks used for handling dangerous pathogens. International declarations have been signed. 38:38 These are tentative steps on the narrow path. Whether they will prove sufficient remains to be seen. The philosopher Daniel Dennett said the singularity was preposterous. 38:49 But he also warned that AI tools we become hyper-dependent on, that is going to happen. And the danger is that we will give them more authority than they warrant. 38:59 I was once the head of social media at a national newspaper in the UK. I saw how the information ecosystem transformed over more than a decade of my career. 39:09 I saw how quickly platforms could reshape what people believed, who they trusted, and what they thought was real. That happened without superintelligence. It happened without AI chatbots or AI agents. 39:22 It happened through giving algorithms and platforms too much authority too quickly, without understanding their insidious and pervasive effects. Cassazade's accumulative hypothesis offers no easy answers. 39:37 It suggests that the real danger lies not in a single catastrophic event, but in the slow erosion of the systems that hold civilization together. By the time the triggering event arrives, the damage may already be done. 39:50 The response capacity may already be exhausted. The coming wave is arriving regardless. The question is whether we walk the narrow path or fall. 40:14 But maybe you could actually unilaterally change the world without having to constantly convince people and beg people and plead with people who are never going to agree with you through technological means. 40:24 And this is where I think technology is this incredible alternative to politics. Our product is used on occasion to kill people. Is this drone targeting or what is this? Targeting of all kinds. 40:37 If you're looking for a terrorist in the world now, you're probably using our government product. 40:42 I want to wait to have kids until we figure out how Neuralink or other, it's called brain-computer interfaces, so other ways for brains to interlink with a computer until they start working. I love killer robots. 40:58 One of the things that is always unfair in this comparison is people talk about how much energy it takes to train an AI model relative to how much it costs a human to do one inference query. 41:10 But it also takes a lot of energy to train a human. It takes like 20 years of life and all of the food you eat during that time before you get smart. 41:18 Citizens will be on their best behavior because we're constantly recording and reporting everything that's going on. And it's unimpeachable. If somebody's committed crime, a more humane form of 41:33 containment of future crime, which is if you say like you now get a free optimist and it's just going to follow you around and stop you from doing crime. 41:49 You've been watching Absolutely Agentic. 42:31 We're a media consultancy and startup focused on AI, and we create weekly YouTube videos and a newsletter we send a few times a week. 42:39 In a world of AI hype, we're trying to strike a balance, looking at a range of opinions to cut through an increasingly complex world. 42:47 The best thing you can do to support us is to sign up to the newsletter in the description or subscribe to the channel. But also don't forget to like and leave us a friendly comment. See you next time. 42:59 It'll be Absolutely Agentic.
