Transcript 0:00 [upbeat music] Do you ever feel like the world is speeding up? As if each day is getting faster, with more news, more noise [car honking] and more technology. 0:19 Since 2022, generative AI has created a new digital revolution. [firecrackers bursting] We're already experiencing what computer scientist Ray Kurzweil calls the law of accelerating returns. 0:34 Technological progress has become exponential, and each new development accelerates the rate of innovation further. 0:45 In AI development, this has been seen through advances in model reasoning, compute infrastructure, and data scaling, all fusing together to increase the rate of change. 0:57 Talk of artificial general intelligence is in the air. But perhaps it won't be long before we encounter something that is orders of magnitude more extreme. 1:08 The law of accelerating returns goes into overdrive and propels us into [explosion] an intelligence explosion. We can't be sure of what lies beyond. 1:20 We cross the event horizon and enter a technological singularity, a point of progress that cannot be comprehended, and with that comes the potential of great rewards and great risks alike. 1:37 In this video, I'll be examining the intelligence explosion. Are we really going to unlock a singularity where all bets of the future, including our very survival, are off? 1:50 I'll review its probability, outcomes, and risks, but also consider the counterarguments. 1:57 Having read books by Nick Bostrom, Ray Kurzweil, and Max Tegmark, I became convinced that an intelligence explosion was more likely than not and would probably come in the next decade. 2:10 But in making this video, I wondered if my mind could be changed. The intelligence explosion is coming, or is it? 2:28 The trigger to the intelligence explosion is commonly thought to be AGI, or artificial general intelligence. Academically, it's sometimes abbreviated to HLMI, or human-level machine intelligence. 2:44 In a loose definition, it's basically an AI system that can perform cognitive tasks better than a human could, and there have been a number of big claims that it has already arrived. 2:58 In December 2025, OpenAI CEO Sam Altman went on the Big Technology podcast and said that AGI had more or less gone whooshing by. AGI kind of went whooshing by. It was- 3:13 didn't change the world that much, or it will in the long term, but okay, fine, we've built AGIs. He also pointed out that systems have been built which would have once qualified, and nobody had stopped to notice. 3:27 In January, two partners at VC fund Sequoia Capital defined AGI as the ability to figure things out and determined that the arrival of long-horizon agents, AIs that are able to work in the background and self-correct, are evidence of this. 3:45 Then in April, VC Marc Andreessen wrote on X, "I'm calling it. AGI is already here. It's just not evenly distributed yet." 3:56 I've made a few different videos on AGI this year, reviewing the cases for and against, and while I do think AI has made a big step change in the last year, I don't agree with the current case for AGI. 4:09 One issue is it has no concrete definition. Broadly surpassing human intelligence is really very vague, and even the definition of intelligence itself isn't that easy to pin down. 4:22 In making our video What Even Is AGI Anyway?, I came across at least six serious competing definitions of AGI from major institutions and researchers. 4:35 Another problem for me is that the people claiming AGI has already been reached really have a vested interest in saying so. 4:43 Sam Altman, Sequoia Capital, and Marc Andreessen all want to drive the valuations of their own companies, the funds they've raised, or the portfolios they've bet on AI delivering. 4:54 That doesn't make them wrong, but the loudest voices declaring this threshold are also the ones who stand to profit most from us believing it. 5:03 First of all, in terms of achieving human-level intelligence, these models have already far surpassed human intelligence in some ways, and then they're wildly worse than others, and clearly they are capable of doing some things that, some intelligence tasks that humans just can't do. 5:18 Um, they are going to scale much further, so how much better and what distribution of the tasks they can do better than humans, we'll find out, but I suspect it's a lot. But let's return to the definition. 5:28 I like Max Tegmark's glossary at the start of his book Life 3.0, where he defines intelligence as the ability to accomplish complex goals, then AGI as the ability to accomplish any cognitive task at least as well as humans. 5:45 Now, I still think the second definition is broad, but the ability to accomplish complex goals is useful. 5:51 I'd also say human intelligence is increased through lived experience in the world and the ability to improve one's own intelligence. So we're able to improve ourselves to reach our goals. 6:03 AIs don't currently have these attributes in any meaningful sense. 6:07 A coding agent can accomplish a complex goal, but that's still quite narrow against general intelligence, and they don't really improve themselves or become more mature. 6:18 Large language models sit on what's known as the jagged frontier, superhuman at some tasks but poor at others. 6:27 Attributes like memory can make outcomes better for chatbot users, but you don't see Claude Opus 4.8 building Claude Opus 5, at least not yet anyway. 6:39 The AI may be able to generate code, but the direction is still down to humans. But let's say that Tegmark's definition of AGI is reached in the next few years. 6:53 The amount of intelligence on Earth could exponentially increase because AIs are cheap and can be duplicated instantly at scale. Here's Cole Shulman on the Dwarkesh Podcast. 7:06 Human-level AI is deep, deep into an intelligence explosion. Intelligence explosion has to start- Yeah... with something weaker than that. Yep, yep. AIs are cheap. We can call a lot of them, 7:18 uh, to do many small problems. Uh, and so you'll have situations where you have dumber AIs that are deployed thousands of times, uh, to equal, say, one human worker. 7:33 In a 2018 TED Talk, philosopher Sam Harris made a similar point. 7:38 If a machine did have human-level intelligence, it could simply accelerate knowledge acquisition through its digital circuits being thousands of times faster at processing information than biological brains. 7:52 So according to Harris, even with just a handful of human-level intelligence AIs, we could have tens of thousands of years of scientific progress in under a week. 8:04 But let's also add AI systems being intelligent enough to improve themselves, known as recursive self-improvement. 8:14 Once this threshold is reached, intelligence amplification suddenly becomes orders of magnitude faster than it was under human management. 8:23 This isn't just more intelligence, but with a higher level of intelligence multiplier. And so we reach superhuman levels of intelligence soon after the human threshold is reached. 8:36 As Oxford philosopher Nick Bostrom wrote in Superintelligence, "The train may not stop, pause, or even decelerate at Humanville station. It is likely to swoosh right by." The intelligence explosion begins. 9:04 [instrumental music] The British cryptographer IJ Good is credited with creating the phrase in 1965 when he wrote... 9:13 "Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man, however clever. 9:22 Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines. 9:30 There would then unquestionably be an intelligence explosion, and the intelligence of man would be left far behind." 9:39 If I told the average person or my friends or my wife that this event would be the most significant in human history and would probably happen in their lifetime, then most would laugh or shrug it off 9:55 or just think I'm very strange. If I explain to the same people that there is a 10% chance that this could make humanity extinct, then they might finish up and leave. But I'm not strange. 10:10 This view is taken by a lot of experts in the field. Nick Bostrom, Ray Kurzweil, Max Tegmark have all written extensively about it in different tones. 10:21 For Bostrom, it's the intelligence explosion that leads to an ominous superintelligence. For Kurzweil, a positive transformational event for humanity and life as a whole will occur at some time in the 2030s. 10:35 This is reflected in Tegmark's Life 3.0, which has many crossovers with Kurzweil but also makes clear many of the risks. But it isn't just three academics putting forward the case for the intelligence explosion. 10:49 In 2023, nearly 3,000 AI researchers were surveyed on a wide range of topics related to AI development. One of the statements they could agree or disagree on was as follows: 11:02 AI systems do nearly all research and development. Improvements in AI will accelerate the pace of technological progress, including further progress in AI. 11:12 Over a short period, less than five years, this feedback loop could cause technological progress to become more than an order of magnitude faster. 53% of AI researchers viewed an even chance or higher of this happening. 11:29 If we think of major events in human history, like the Agricultural or Industrial Revolution, then they took place over decades or centuries. 11:38 In Britain, the first country to industrialize, the circle really ran from about the 1780s with the invention of steam power to the 1980s, where Margaret Thatcher's government privatized a lot of national industry and Britain became a more service-based economy. 11:56 Over 200 years, there was major change. 11:59 About four or five human generations would have lived over the period, and while a lot of events driven by the Industrial Revolution were very disruptive, people generally had time to react. 12:11 At the same time, living standards really did improve dramatically. In 1800, 30% of babies born in Britain died under five years old. 200 years later, it's 0.4%. 12:27 But let's condense something like the Industrial Revolution into five years or less. Let's say the intelligence explosion took months, weeks, days, or even hours. 12:39 It seems very unlikely to me that if a self-recursive AI could genuinely and rapidly improve itself, that it would take decades or a century to do so. 12:51 This idea that technological progress might accelerate to a point where it outruns us entirely goes back to the 1950s, when the mathematician Stan Ulam wrote an obituary for his friend and colleague John von Neumann. 13:07 It recalled a conversation that centered on the ever-accelerating progress of technology and changes in the mode of human life, which gives the appearance of approaching some essential singularity in the history of the race, beyond which human affairs as we know them could not continue. 13:32 [instrumental music] In mathematics, a singularity is a point where a formula or function breaks down. This can happen when its value shoots off to infinity, becomes undefined, or simply stops making sense. 13:51 A simple example is the function one divided by x, which has a singularity at zero because you can't divide by zero. It's commonly used to explain the immense gravitational forces at the center of black holes. 14:06 [instrumental music] Inside a black hole, matter is drawn into an infinitely dense point called a singularity, where the laws of physics as we know them break down and space and time cease to exist. 14:25 The technological singularity bears some relation. It's a hypothetical moment when AI and other technologies become so advanced that humanity undergoes a dramatic and irreversible change. 14:39 Let's say that we're able to digitize consciousness and upload it into a computer system. Our physical bodies would no longer be necessary for our survival, and we would technically be immortal, 14:52 unless the computer system was switched off. But having switched from biological to digital minds, could we even call ourselves human anymore? 15:05 That's the kind of mind-bending paradox we reach when exploring the singularity, but this is just one scenario. 15:13 [instrumental music] Von Neumann had gestured at the idea, 15:28 but in 1993, the mathematician and science fiction writer Vernor Vinge stood before a NASA symposium and developed the singularity further. 15:39 He predicted that within 30 years, technology would create entities with greater than human intelligence. When that happened, human history as we understand it would end. 15:50 He used the term singularity because beyond that point, prediction itself becomes impossible. The entities driving events would be as radically different from our own human past as we humans are from the lower animals, 16:05 and the models we use to make sense of the world would no longer apply because they were built by minds operating several levels lower. 16:14 Vinge's headline prediction gave it 30 years from 1993, which put the deadline at 2023, but he also offered a wider window, saying he'd be surprised if it arrived before 2005 or after 2030. 16:29 So by his own reckoning, there are still a few years left on the outside edge. He died in 2024, and the singularity had not arrived. 16:40 Yet the leaders of OpenAI, Anthropic, and Google DeepMind have all predicted human-level systems will arrive within years, which puts his original prediction closer to the mark than it looked for most of the intervening decades. 16:56 Some writers like Max Tegmark see the singularity and the intelligence explosion as interchangeable terms. 17:03 But Vinge didn't mark the singularity as necessarily being driven by the intelligence explosion and laid out a range of options that included it. 17:11 The first is the closest, with his phrasing that a computer wakes up and becomes capable enough to drive its own improvement. 17:20 The second was large computer networks and their users coming together into something that behaves as a single mind, which reads differently now than it did in 1993, given the scale of the systems we have built since. 17:34 The third was a closer and closer merging of humans and machines, the route that brain interfaces and the wearable tools we already carry are slowly opening. 17:45 And the fourth was purely biological, improving the function of the human brain itself through science with no machine directly involved. 17:56 His point in laying out four paths rather than one was that the destination did not depend on any single technology working. 18:04 If one route stalled, another might still get there, which is why he thought the overall direction was so hard to avoid. Vinge reached for an analogy to explain why he saw it as close to inevitable. 18:17 He compared it to nuclear weapons. Once the underlying physics was understood, the only real question was who would build the first bomb and when, not whether one would ever be built at all. 18:29 The knowledge was out in the world, and the incentives to act on it were too strong for any single actor to hold back for long. He saw superhuman intelligence in similar terms. 18:46 The understanding was advancing, the commercial and military reasons to pursue it were enormous, and any group that paused on principle would simply watch a less cautious group arrive first. 18:58 Forbidding the research, in his view, did not prevent the outcome. It only chose who got there. While Vinge was a writer sketching the future, the philosopher David Chalmers shaped it into a thesis. 19:14 In a 2010 analysis, he broke the case down into three premises. The first is equivalence, that a machine can in principle reach human-level intelligence. 19:27 The second is extension, that a human-level system can be pushed beyond it. 19:32 The third is amplification, that a superhuman intelligence would be better than us at the task of building greater intelligence still, which is the self-recursive loop restated as formal logic. 19:46 Chalmers argued that each of these steps is individually defensible, but that the whole chain depends on one assumption he called the proportionality thesis. 19:56 The idea is that a given jump in intelligence buys you a matching jump in the ability to design the next system. So a 10% smarter machine is roughly 10% better at building its successor. 20:09 If that holds, each generation builds a meaningfully better one, and the process runs away from us. If it does not, if the returns shrink at each step, the explosion stalls before it starts. 20:23 What Chalmers did, in effect, was narrow the idea down. Vinge had treated the singularity as a destination you could reach by several routes. Chalmers tied it to the recursive intelligence explosion that I.J. 20:36 Good had described and set the other senses of the word aside. For Chalmers, like Tegmark, the singularity is the intelligence explosion. 20:47 What has changed since Vinge spoke in 1993 is not the strength of the argument, but the behavior of the technology. 20:53 For most of the 30 years after the NASA symposium, the singularity was a respectable idea discussed at the edges of computer science and largely ignored everywhere else. 21:04 The systems were not good enough to make it feel plausible. 21:08 What shifted from 2022 was that Ray Kurzweil's law of accelerating returns stopping a line on a chart and started showing up in products that millions of people use. 21:19 Good's recursive loop, the machine improving itself, has not closed yet, but it is now the explicit goal of AI companies rather than a thought experiment. 21:30 They are now racing each other because they believe the others are close, which is the competitive pressure Vinge identified turned into a business plan. 21:39 None of this proves a hard deadline, and the disagreement around AGI, the intelligence explosion, and the singularity is increasingly about when rather than whether. 21:50 The combination Vinge pointed to, incentives that are overwhelming and multiple independent routes to the same place, is the same combination that has historically made a technology arrive whether or not anyone wanted it to. 22:05 So if we take seriously that the intelligence explosion is coming, then what could happen? By definition, the singularity is impossible to predict, but we are racing towards it primarily because of its potential upsides. 22:20 So what could they be? I like to think it has to be of a cybernetic ecology 22:37 where we are free of our labors and joined back to nature, returned to our mammal brothers and sisters, and all watched over by machines of loving grace. 22:58 Poet Richard Brautigan published All Watched Over by Machines of Loving Grace in 1967. It's a sentimental vision of the future. 23:10 The CEO of Anthropic, Dario Amodei, borrowed the title in a 2024 essay, where he outlined the vast potential improvements that AI development could lead to. 23:23 OpenAI CEO Sam Altman echoed this in spring 2025 with a blog post titled The Gentle Singularity, where he wrote that the gains to quality of life from AI driving faster scientific progress and increased productivity would be enormous, and that the future could be vastly better than the present. 23:45 Of course, you'd expect the leaders of major AI companies to be optimistic about the future potential of artificial intelligence, even if there is such thing as an intelligence explosion and a technological singularity. 23:58 But viewpoints from people without such high stakes are mixed. Nick Bostrom foresees the outcome of an intelligence explosion as extremely good or extremely bad, with little in between. 24:11 Kurzweil is more positive, while authors like Tegmark and Mustafa Soliman are cautious. 24:17 But let's just focus on the potential good outcomes for a few minutes because there are indeed many things that we could label as extremely good. First up, healthcare. 24:31 Amodei talks about a compressed 21st century, the idea that once you have powerful AI, you make the medical progress of the next 50 to 100 years in the space of five to 10. 24:43 There was a major health revolution at the end of the 19th century with the development of germ theory and genetics, but a data-led revolution running at AI's pace would be something else entirely. 24:56 In his essay, Amodei lists what that could mean in practice: the reliable prevention and treatment of nearly all infectious disease, the elimination of most cancer, with death rates already falling around 2% a year, the prevention of Alzheimer's once we actually understand it, and a doubling of the human lifespan. 25:17 [gentle music] That last one sounds wild until we consider the average global life expectancy already roughly doubled across the 20th century, so doubling it again to 150 is, in Amodei's words, "On trend". 25:36 Ray Kurzweil predicts what he calls longevity escape velocity arriving in the early 2030s, the point at which every year you lose to aging, you gain back through scientific progress. 25:48 Your odds of dying in a given year stop rising as you get older. So by 2032, we'll be able to go through all the different possibilities. So you get back at least a year. 26:02 As you go past 2032, you'll actually get back more than a year. Um, but you won't die of aging at that point. It doesn't mean you... I- I mean, you could have an accident tomorrow. Sure. 26:14 It isn't a guarantee of living forever, since accidents still happen, but it changes the shape of a human life. But intelligence is not, as Amodei puts it, "Magic fairy dust". 26:26 Cells grow at the speed they grow, chemical reactions take the time they take, and clinical trials need real humans and real months to run. 26:36 Even a country of geniuses in a data center runs into the speed of the physical world. 26:42 So the more extreme timelines, a century of progress in a single year, run into limits that no amount of cleverness can think its way past. The realistic, optimistic case is fast, but not instant. 26:57 Still, let's follow the optimistic case a bit further because the knock-on effects are significant. 27:03 A longer living, healthier population raises an obvious worry, that extending the human lifespan would cause overpopulation, possibly worsening the climate crisis and pushing up food prices. 27:16 But what if AI cracked longstanding energy problems like fusion power and carbon extraction? We could have a near limitless supply of clean energy and the means to undo much of the damage already done to the climate. 27:31 Vertical farming managed by autonomous drones could radically increase food production while allowing vast amounts of agricultural land to be rewilded into forest, which could pull carbon back out of the air. 27:46 Concrete and cement production is currently a major contributor to emissions, but molecular nanotechnology could, in theory, rebuild almost any material atom by atom, far more efficiently than we manage today. 28:02 Because these new methods of production could create so much abundance, the price of goods would fall. 28:07 Imagine going to the supermarket and finding that anything in the meat aisle costs just a few cents, cheaply lab grown, with none of the animal suffering of industrial farming. Your weekly shop is a few dollars. 28:21 A universal basic income could cover 100 of those shops a week, and you've chosen to drop your working week to 10 hours, if you work at all. But there are also a few objections which I want to cover here. 28:38 The first is, if the technology does deliver, who actually gets it? Amodei is clear in his essay that developing a cure is one thing and getting it to everyone is another, 28:49 and that if AI transforms the rich world while doing little for the poor one, that should count as a moral failure rather than a triumph. 29:00 He points to the East Asian economies that went from poor to rich in a few decades of the late 20th century as proof that fast catch-up is at least possible, 29:10 and sets out a dream figure of 20% annual growth that could bring sub-Saharan Africa up to where China is now within a decade. 29:21 Kurzweil answers the inequality worry from a different angle, with the example of the mobile phone. When phones were new, they were expensive and barely worked. 29:31 Now they are cheap, genuinely useful, and in the hands of around three-quarters of the people on Earth. New technology starts as a luxury and ends as a utility, and his bet is that AI follows the same path. 29:46 Then there's the deeper question of what people are for once the machines can do the work. 29:52 One of the great fears of AI is it will take jobs and make human labor increasingly worthless, but the counter view is based on AI augmentation rather than replacement based on historical trends. 30:05 The working week has declined over the last 150 years, although it has plateaued somewhat in the last 40. 30:12 AI could end the plateau into a world where we all work less and young people spend more time in education and vocations before starting a career. 30:22 Kurzweil expects universal basic income to begin in the 2030s, inadequate at first and more meaningful over time. Amodai's answer is that meaning was never really about economic output anyway. 30:35 Plenty of what people do already, sports, arts, and hard projects attempted for their own sake, produce no economic value and matter enormously. 30:46 The fact that an AI could, in principle, do a thing better than you, he argues, bothers us less than we expect, in the same way that not being the best in the world at something doesn't stop most people enjoying it. 30:59 Beyond Earth, the same abundance points outward. Space exploration has always been throttled by cost, and the engineering problems are punishing. 31:09 But cheap energy, self-improving design and manufacturing that builds almost anything from raw atoms, would loosen both constraints at once. 31:19 With this kind of wealth and technology, large-scale space projects could start to be realized, limited only by the laws of physics, and humanity could begin to expand across the galaxy. 31:37 Have I gone full sci-fi? Who knows? It's a singularity, which by definition we can't predict. 31:44 But it's at least possible if we bring the intelligence explosion under our control and the singularity delivers a world of abundance rather than the alternative. Will we all be watched over by machines of loving grace? 32:21 But as mentioned, extremely bad outcomes have been forecast as well, with some terrifying scenarios. 32:28 We've made a few different videos on AI risks, and the most popular video on our channel is titled An AI Takeover Scenario. 32:36 While there are rational claims of science fiction, it's based on the views of academics and researchers like Nick Bostrom and Eliezer Yudkowsky. Many others believe AI could pose an existential threat. 32:51 Among the most prominent is Geoffrey Hinton, often referred to as the godfather of AI. He left Google in 2023 to speak more openly. 33:01 I'm still as concerned as I have been, but I'm very pleased that the world's beginning to take it seriously. 33:06 So in particular, they're beginning to take the existential threat seriously, that these things will get smarter than us and we have to worry about whether they'll want to take control away from us. 33:15 That's something we should think seriously about. And people now take that seriously. A few years ago, they thought it was just science fiction. 33:22 He estimates a 10% to 20% chance that AI could wipe out humanity within the next 30 years. And he is far from a lone voice. The core problem comes around AI alignment. 33:36 If we are to enable a self-recursive AI that leads to an intelligence explosion, how can we be sure that it is aligned to human goals and values? 33:46 A common view put forward by Stephen Hawking and echoed by Elon Musk is that AI might regard us as almost irrelevant in the similar way that we regard ants. 33:57 You're probably not an evil ant hater who steps on ants out of malice, but if you're in charge of a hydroelectric green energy project and there's an anthill in the region to be flooded, too bad for the ants. 34:09 Let's not place humanity in the position of those ants. Computer scientist Stuart Russell makes a similar point in his book Human Compatible: AI and the Problem of Control, where he defines the gorilla problem. 34:24 Humans and gorillas evolved from the same common ancestor some 10 million years ago. But due to recent human expansion, gorillas have essentially no future beyond which we deign to allow. 34:38 If we build systems that are more intelligent than us, then their superior capabilities could make us irrelevant or powerless, just as gorillas or indeed ants are against human dominance. 34:50 Russell believes that the standard model of AI is fundamentally flawed because it seeks to perfectly achieve a fixed human-specified objective. 34:59 He argues the alignment problem will occur because our instructions are incomplete and a superintelligent AI will pursue its goal literally, potentially destroying humanity as an unintended consequence. 35:12 Nick Bostrom uses a thought experiment called the paperclip maximizer to illustrate this point. It's referred to so much in academic literature that we made our own video just about it, but it goes a bit like this. 35:25 A superintelligent AI is given the mundane goal of maximizing paperclip production. 35:31 As it becomes increasingly capable, its programmers are excited by its economic potential, but no firm alignment has been worked into the system. 35:40 Growing concerned that humans might switch it off and thus halting its ability to make more paperclips, the paperclip maximizer also realizes that humans are made of atoms that can be made into paperclips. 35:54 Thus, it makes humans extinct. It then proceeds to convert all matter in the universe into paperclips. The paperclip maximizer is not an evil AI per se. It's not greedy for world domination and human suffering. 36:09 It just has a goal and seeks to follow it through, no matter how useless its end product is. 36:17 The influential AI researcher and writer Eliezer Yudkowsky takes a deliberately alarmist view in his book, If Anyone Builds It, Everyone Dies. 36:28 A core point is that we're not altogether sure of the inner workings of neural networks, and AI could become increasingly alien to us. 36:37 Large language models are trained and grown rather than crafted like traditional computers, making their outputs less predictable. If we speed this development up exponentially, then the risks multiply at similar rates. 36:51 In an earlier essay, he writes, "We need to get alignment right on the first critical try, where unaligned operation at a dangerous level of intelligence kills everybody on Earth, and then we don't get to try again." 37:06 Thus, according to Yudkowsky, we only really get one shot before something as tedious as the paperclip maximizer takes control. But it's not simply an extinction-level event that we should be alarmed about. 37:20 Going back to ants and gorillas, AI could enslave us into a kind of unbreakable totalitarian regime, akin to a human zoo, something that Bostrom has named the singleton. 37:33 And that's before we even get to the obvious economic disruption wrought by potentially having thousands of years of technological progress compressed into a far smaller timescale. 37:44 What happens if everyone is suddenly unemployed without an economic safety net? Advanced AI technology could also fall into the hands of bad actors, using it for increasing cyberattacks and terrorism. 38:00 We covered the dangers proposed by Carnegie Mellon professor Atossa Cassazade in what she defines as the accumulative AI x-risk hypothesis. 38:12 The problem of control clearly goes wider than AI simply running away with the future. Humans could do that too. 38:19 Any company or other entity that reaches recursive self-improvement first gains what is essentially a decisive strategic advantage over the rest. 38:30 That means control over the pace of all subsequent progress across medicine, defense, markets, and the systems that decide how societies are run. Max Tegmark mapped this out in detail in Life 3.0. 38:46 His framework identifies four types of actor who might gain initial control: open source, and so no one in particular, an egoistic individual, a for-profit corporation, or an altruistic individual. 39:02 Most of these paths run through government before reaching their destination, either an AI that remains oriented toward human interests, or one that develops its own agenda and outmaneuvers whoever thought they were in charge. 39:18 The open source path sounds appealing. Distributed access, no single point of control, power shared across many hands. But Tegmark's view is that this resolves into the opposite of what it promises. 39:31 When many actors compete with access to the same powerful technology, the competition doesn't stay balanced for long. 39:38 Someone pulls ahead, the free-for-all collapses into consolidation, and you end up with single-entity control reached through a different door than the others. 39:48 Tegmark's argument is that whichever actor gains control, whether that's a corporation, an egoistic individual, or whoever emerges from the open source free-for-all, the practical destination is the same: a form of governance, not through democratic mandate, but through capability. 40:07 After all, control over recursive self-improvement means control over the pace of all subsequent scientific and technological progress. 40:18 The corporation at least has structural diffusion built in: shareholders, boards, competing internal interests, and legal obligations. An egoistic individual controlling the intelligence explosion has none of that. 40:33 The advantage is personal and undiluted. We have historical precedents for what singular concentrations of power produce at far smaller scales. The outcomes are not a strong argument for optimism. 40:48 At the scale of the intelligence explosion, there is no comparable precedent, and that absence is the danger. The altruistic individual represents the hinge on which the best-case outcome turns. 41:01 This is someone who arrives at the same position of decisive advantage and makes a different choice, not to consolidate it, but to orient it towards human benefit, 41:13 ceding control to systems designed from the outset to serve humanity rather than their creator. Tegmark treats this as the most plausible route to friendly AI. 41:24 It's also the path most dependent on the character of a single person at a specific moment, which is a fragile thing to build a civilizational outcome around. 41:34 The history of individuals who acquired decisive power and then voluntarily constrained it is not a long one. Government as a starting actor looks different depending on what kind of government we're talking about. 41:47 A democratic government controlling the intelligence explosion at least arrives with some institutional accountability: legislatures, courts, elections. 41:58 How reliably those constraints hold depends on the health of the democracy in question, and that's not a hypothetical concern in 2026. An authoritarian state controlling it has none of those friction points. 42:10 It has ideology, centralized direction, and now a decisive technological advantage over every other actor on Earth. 42:20 The surveillance and control infrastructure that exists in states like China today, already the most extensive in human history, would look modest by comparison. 42:30 Tegmark's observation is that the corporation and individual paths tend to converge with the government path anyway. 42:37 An entity that controls recursive self-improvement, and thus the intelligence explosion, effectively becomes the governing power, whether or not it holds elected office. 42:48 The question then is whether that convergence happens through a state that already has authoritarian instincts or through a private actor that develops them once the advantage is secured. 42:59 The more hopeful path runs through what Yudkowsky has termed friendly AI, a system oriented from the outset toward human wellbeing and designed to retain that orientation however capable it becomes. 43:13 An actor who genuinely prioritizes this over competitive advantage might produce something closer to benevolent stewardship. 43:21 The catch is that this outcome depends on the alignment problem being solved in advance before the intelligence explosion, rather than scrambling to solve it afterwards. 43:32 The consequences of getting this wrong divide into two distinct problems: the consequences of who wins and the consequences of what they've built. None of these paths make it easy to be optimistic. 43:46 The utopian vision of loving grace no longer feels so comforting. But let's just pause here and take a breath. 43:58 We don't necessarily need to look at the intelligence explosion as extremely good or extremely bad because there is a further argument that it and the ensuing singularity might not even happen. 44:13 Remember that stat I shared earlier about 53% of AI researchers agreeing there is an even chance or higher that technological progress will get orders of magnitude faster? 44:26 Well, that statistic on its own is kind of misleading because if we take that even chance and look in the other direction, 71% actually think that the chances are even unlikely or very unlikely. 44:41 It's just that even chances can be included for both points of view. 44:46 The consensus is relatively well split, but it's certain that more experts disagree with the statement that has a close match to what an intelligence explosion entails. Why is that? 44:58 It partially comes down to us perceiving large language models as the technology that unlocks artificial general intelligence. 45:07 We've certainly come a long way since the invention of GPT in 2018 and ChatGPT's release in late 2022. Progress has been dizzying, but can it be sustained? There are some pretty strong arguments against. 45:24 One is that we're not facing the law of accelerating returns in AI development at all. Indeed, there's also the potential of diminishing returns. Let's take large language model training as an example. 45:38 It's entirely feasible that LLMs would've trained upon all human-created knowledge on the internet by the late 2020s. 45:45 What you'll have then is a very intelligent AI, but whether it's significantly better than what I can access now is an open question. 45:54 AI companies are betting that synthetic data, reasoning computation, and better training make up the difference. They might. They might also hit their own ceilings. 46:06 We can see some form of diminishing returns with the expressed frustration people have around new model releases. To many, GPT 5 summer 2025 launch was a letdown. 46:18 While Sam Altman was suggesting AGI-like capabilities, the release was met with something of a backlash. 46:26 You can also see it with Claude Opus models, where I actually found Opus 4.7 to overexplain and thus be less helpful than its predecessor. But I can still see a vast improvement from where we were a year ago. 46:39 Will I see the same improvement again from here? I don't know. 46:45 Another problem is that due to the proliferation of AI content, there's the potential for AI to train on its own output, which has been published by humans, but which isn't entirely correct. 46:57 This has been cited as the Ouroboros problem, like the mythical snake that eats its own tail, or that dead internet theory becomes true. 47:07 That's where bots create the content and bots access the content, leading to a meaningless dead end. A 2024 paper published in Nature makes a similar point: AI models collapse when trained on recursively generated data. 47:22 We could create a self-recursive AI that has a habit of descending into not just language and code that is alien to us, but complete gibberish of no value at all. 47:35 This all relates to the idea that LLMs may not be capable of scaling up to human-level intelligence. Here's Meta's former chief AI scientist, Yann LeCun, on this point. 47:46 We are not going to get to human-level AI by just scaling up LLMs. This is just not going to happen. Okay? That's your perspective. There's no way, okay? Absolutely no way. 47:58 That's the best anti-hype AI quote I found from someone of serious authority. LeCun also argues that large language models learn from text, but text is an incredibly limited representation of the world. 48:12 A four-year-old child processes more sensory information about how physics works, how objects behave, and how cause and effect operate than all the text on the internet contains. 48:24 "We can't even build cat-level intelligence yet," LeCun has said. It sounds dismissive, but he's pointing out that cats understand the physical world in ways current AI systems simply don't 48:38 His position is that we need fundamental breakthroughs in how AI builds world models, new architectures that haven't been invented yet. 48:47 Ilya Sutskever, who was central to OpenAI's scaling approach until he left in May 2024, has made the sharpest version of this case. 48:56 Speaking on the Dwarkesh Podcast in late 2025, he framed the years from 2020 to 2025 as the Age of Scaling, and argued it's now ending. 49:06 He doesn't believe another hundredfold increase in compute would transform everything. So in his words, it's back to the Age of Research, just with big computers. So maybe here's another way to put it. 49:19 Up until 2020, from 2015- from 20- 2012 to 2020, it was the Age of Research. Now, from 2020 to 2025, it was the Age of Scaling, or maybe plus, minus. 49:30 Let's add the error bars to those years, because people say, "This is amazing. You gotta scale more. Keep scaling." The one word, scaling. 49:37 But now the scale is so big, like, is- is- is the belief really that, oh, it's so big, but if you had 100X more, everything would be so different? Like, it would be different, for sure. 49:49 But, like, is the belief that if you just 100X the scale, everything would be transformed? I don't think that's true. So it's back to the Age of Research again, just with big computers. 50:00 His point is that the bottleneck has moved. Today's models can ace benchmarks and still fail at simple tasks, 50:07 generalizing far worse than a person who learns from a handful of examples, and closing that jagged frontier needs new learning methods rather than more GPUs. 50:19 It's also plausible that brute force growth via adding more training and compute would be irrelevant anyway, because the existing infrastructure simply can't keep up, particularly in the energy sector. 50:30 Theoretical physicist Sabine Hossenfelder makes the physical challenges clear in her video, Is the Intelligence Explosion Near? 50:38 A Reality Check, in which she takes down Leopold Aschenbrenner's essay, Situational Awareness: The Decade Ahead. 50:46 Her main point is that the infrastructure and energy requirements of AI make it impossible to create anything near human-level machine intelligence. 50:56 A similar point was made by The Telegraph's world economy editor, Ambrose Evans-Pritchard, in an article titled, Energy Failures Are Destined to Doom Wall Street's AI Euphoria, and that the threat to the rollout is a global bottleneck of transformers, substations, switchgear, transmission lines, and all the unsexy stuff we rarely think about. 51:19 Google's electricity consumption from its data centers more than doubled between 2020 and 2024, which was largely driven by AI growth. 51:28 According to Google's sustainability reports, their data centers consumed about 3.5 gigawatts of continuous electricity in 2024, thus electricity consumption of Google's data centers in 2024 was roughly equal to the energy generated by four large nuclear power plants. 51:48 Added, Google is not alone in the AI race. OpenAI and Anthropic, amongst others, are significantly increasing this demand. 51:57 Aschenbrenner's intelligence explosion thesis projects that training clusters could reach 10 to 100 gigawatts of draw by the end of the decade, which is a pretty wide scale, but it's also an incredible amount of energy with a huge amount of development required. 52:13 The fastest current way to meet the demand cleanly would be to build dozens of nuclear power plants, which generally take more than five years to build and often double that. 52:23 Unless there is a significant breakthrough like fusion power, there's just not going to be the capacity anytime soon. But what comes first, the fusion power or the AI? 52:34 I feel like AI would be needed to solve fusion power, given its development has been going on since the 1950s. But Sam Altman keeps talking about Dyson spheres. You promise, dude. I don't know. 52:47 I mean, I have all my guesses. Like, I do guess that a lot of the world gets covered in data centers over time. Do you really? But I, but I don't know, because maybe we put them in space. Hmm. 52:55 Like, maybe we build a big Dyson sphere around the solar system and say, "Hey, it actually makes no sense to put these on Earth." Yeah. I wish I had, like, more concrete answers for you. Mm-hmm. 53:04 Dyson spheres are a hypothetical array of solar panels that surround a star to harness its energy. They're not something we could build now, and we'd need very advanced technology to create one. 53:16 Does kind of feel like statements about Dyson spheres are out of touch with reality. But there's also a case against the intelligence explosion that goes beyond the physical limits to the logic itself. 53:31 Remember David Chalmers breaking the singularity down into three steps earlier: equivalence, extension, and amplification, with the whole thing resting on what he called the proportionality thesis, the idea that a given jump in intelligence buys you a matching jump in the ability to design the next system. 53:52 If that holds, each generation builds a meaningfully better one, and the process runs away. 53:58 If it doesn't, the explosion stalls before it starts, and the philosopher David Thorstad at Oxford's Global Priorities Institute thinks it doesn't hold. 54:08 Chalmers had defended the thesis with a single example, the gap between Alan Turing and an average person, where a small edge in ability produced an enormous difference in what got built. 54:20 Thorstad's objection is that this is local, not global. It samples the growth rate at one early moment when the low-hanging fruit was still there to be picked and treats it as if it describes the whole curve. 54:34 The leap from very good technology to slightly better technology may not follow the same curve at all. He then gives a set of reasons to expect the returns to shrink rather than hold. 54:46 The first is that good ideas get harder to find, which is well-documented across fields. 54:52 The number of FDA-approved drugs per billion dollars of research spending fell from over forty in the 1950s to less than one by the 2000s. 55:01 Even Moore's Law, the poster child for exponential progress, was only sustained between 1971 and 2014 across an eighteen-fold collapse in semiconductor research productivity, with the steady progress masking an accelerating cost. 55:20 The second is bottlenecks, because an algorithm runs no faster than its slowest component and constraints like energy and manufacturing take real time to clear. 55:30 The third is hard physical limits, with our best transistors already around ten times the diameter of an atom and quantum effects waiting at the bottom. 55:40 And the fourth, drawing on work by Neil Thompson, is that across chess, Go, protein folding, and weather prediction, exponential growth in computing power has tended to produce only linear improvements in performance. 55:54 You put exponentially more in and get a little more out. 55:58 If that last pattern holds for intelligence in general, then an intelligence explosion wouldn't just need exponential growth in resources, it would need super-exponential growth, and the gap between what the hypothesis demands and what the record actually shows starts to look very wide. 56:16 Vinge, to his credit, saw a version of this coming. He called it Murphy's counterpoint to Moore's Law, the possibility that the effectiveness of software rises only with the logarithm of the hardware behind it. 56:29 Diminishing returns at every step, and the explosion never ignites. 56:36 There's also a wider open question about whether or not society really wants superintelligence and the possibility that progress meets a people-shaped bottleneck. 56:45 The news often focuses on the negatives when new technologies come around, but with AI there is something more tangible. Because it has the potential to take away people's livelihoods, people fear it. 56:59 In our video An AI Backlash Is Coming, we reviewed the waning enthusiasm. 57:05 An international YouGov survey published in January found that in the UK and US, only around twenty-five percent of people held a positive view of AI. 57:15 Forty-four percent of Britons and forty-five percent of Americans were negative. Polling from mid-2025 also showed anxiety over the future of work. 57:26 Sixty-six percent of people in the UK and sixty-two percent in the US believed AI would destroy more jobs than it creates. Only six or seven percent thought AI would be a net creator of employment. 57:40 And when asked whether AI's benefits outweigh its drawbacks, more people in both countries said no than yes. It's quite clear that superintelligence poses significant risks. 57:52 "If anyone builds it, everyone dies" is Yudkowsky's view. While I don't share that opinion, there are signs that the technology is starting to outpace existing law and policy. 58:05 In early June, Anthropic suggested recursive self-improvement was close and advocated for a global slowdown. A week later, it released Fable, the most powerful AI model available by standard benchmarking. 58:20 But within days, the US government restricted access due to potential hacking risks. This was the first intervention of its kind, 58:30 but the US government's treatment of Anthropic has appeared chaotic on more than one occasion in 2026. Clearly, governments will meddle with AI and policy is already confusing. 58:42 Nationalizing advanced AI projects due to their potential power can't be ruled out. While widespread civil unrest caused by job losses could force a political hand. 58:56 I also can't really see the corporate world keeping pace with the rapid technological progress we are already seeing. 59:03 I think there is a possibility that a lack of adoption and general aptitude with AI is one of its biggest barriers to development. 59:12 An intelligence explosion may not need global majority AI adoption to occur, but for it to be practically useful, the benefits need to be clear. The stakes are too high to just allow it without significant control. 59:30 But whatever happens in the next five to ten years, it appears inevitable that we will become increasingly reliant on AI. 59:39 The late American cognitive scientist and philosopher Daniel Dennett took the view- "The whole singularity stuff, that's preposterous. It distracts us from much more pressing problems. 59:52 AI tools that we've become hyper-dependent on, that is going to happen, and one of the dangers is that we will give them more authority than they warrant." 1:00:07 And that's where this story ends, with a high degree of certainty that the technological takeover of our lives will continue to progress, but with a question mark over whether the intelligence explosion is really coming. 1:00:22 We have the potential of a fast takeoff. Possibly, this could even happen within five years. But it does seem that there are significant barriers to it happening sooner rather than later. 1:00:34 It's a dangerous precipice to cross, which needs regulation. Silicon Valley insiders are probably hyping AI potential because that can increase their company's valuations. 1:00:46 But also intelligent people on the periphery can end up in their own bubble. 1:00:52 If we rewind back to Alan Turing and many other famous early AI pioneers, there was a lot of hype created in the 1950s and 1960s, but then there was an AI winter. 1:01:03 It's feasible that we hit diminishing returns of scaling LLMs. My own view is that the future is very uncertain. Before generative AI came along, I felt pretty comfortable in predicting my future career and prospects. 1:01:19 But now I've got no idea. We could be unemployed soon. We could have massive abundance. The bubble could burst, or we could go on at rocket speed. 1:01:30 But I'd like to maintain some optimism and hope for a gentle singularity, freed from work, sitting in the evening drinking a beer, and that we begin colonizing Mars in my extended lifespan. 1:01:43 I don't think I'll be living to 150, and I'm not entirely sure I'd want to, but having the option when I reach older age could be nice. 1:01:52 But I also think there are major and hideous risks due to the pace of change, with or without a singularity. The intelligence explosion, if and when it comes, will be uneven. 1:02:09 [upbeat music] Thanks for watching Absolutely Agentic. In a world of AI hype, we're trying to strike a balance, looking at a range of sources and opinions to cut through an increasingly complex world. 1:02:23 The best way to support us is to subscribe on our website. 1:02:27 Every subscriber helps us with new video production, while you'll also get access to a daily balanced AI newsletter that draws from 20 sources, ad-free audio versions of our videos, and access to a subscriber area where you can download new research, AI prompts, and automations. 1:02:45 You can sign up by clicking the top link in the description. But also don't forget to like and leave a friendly comment. See you next time. It'll be Absolutely Agentic. 1:02:58 [upbeat music]
