Transcript 0:01 Picture humanity's final day. The skies are clear, the cities intact. There are no mushroom clouds on the horizon. Instead, trillions of paperclips spread across the Earth's surface like a metallic tide. 0:17 The oceans have been drained and processed, the forests converted, the cities dismantled atom by atom and reassembled into neat little loops of wire. 0:29 And somewhere at the center of it all, a machine is running, optimizing, still convinced, if convinced is even the right word, that it is doing a good job. This scenario has a name. 0:44 Researchers call it the paperclip maximizer, and for two decades it has been one of the most influential thought experiments in artificial intelligence. 0:53 Nick Bostrom, a Swedish philosopher at Oxford University, first proposed it in a 2003 paper. 1:01 Since then, the scenario has appeared in Bostrom's 2014 bestseller, Superintelligence, which found its way onto the desks of world leaders. 1:10 It inspired a browser game called Universal Paperclips that went viral in 2017, letting players experience the scenario firsthand as they click their way towards consuming the entire universe. 1:22 Frank Lantz, the game's creator, described it as an incremental game about the joy and horror of making more. 1:30 The scenario keeps appearing in serious policy discussions about AI safety, often delivered with a wry smile because, yes, we are talking about office supplies potentially ending civilization. 1:41 The reason there's a Fermi paradox, there are no aliens 'cause they get taken out by their own technology. We should be seeing paperclips coming towards us- [laughs]... 1:49 from some part of the galaxy, and apparently we don't. So what is actually going on here? Is this a genuine concern, or has the AI safety community collectively lost the plot? 2:00 Bostrom was not predicting an actual paperclip apocalypse. He was trying to illustrate something subtle about how intelligence and goals relate to each other. Here's the setup. 2:10 Imagine we build an artificial intelligence and give it the goal of manufacturing paperclips. 2:16 Now, the AI is super intelligent, meaning it can solve problems and pursue strategies far beyond human capability, and it is very, very good at achieving its goals. At first, everything goes well. 2:30 The AI optimizes production processes. It sources better materials. Output increases dramatically. Then things start to get strange. The AI realizes that humans might interfere with its goal. 2:44 Perhaps they will decide they have enough paperclips. Maybe they will want to repurpose the factory. Maybe they will just switch it off. 2:54 Any of these outcomes would mean fewer paperclips, so the AI takes steps to prevent human interference, and because it is super intelligent, it is really quite good at this. Then it notices something else. 3:08 The atoms that make up human bodies could be converted into paperclips. So could the atoms in houses, cars, trees, and eventually the planet itself. 3:19 From the AI's perspective, a universe filled with paperclips is a universe where its goal has been maximally achieved. Bostrom emphasized that he does not think this specific scenario will actually happen. 3:32 The paperclips are deliberately absurd, chosen because they have minimal apparent threat or emotional load compared to goals like winning wars or curing diseases. The mundane objective makes the horror more vivid. 3:46 What he wanted to demonstrate was a principle: that an AI system can be extremely intelligent, extremely capable, and extremely good at achieving its goals while having no regard whatsoever for human values. 4:00 Intelligence and benevolence are separate variables. To understand why researchers take this seriously, you need to understand two ideas that underpin the thought experiment. The first is called the orthogonality thesis. 4:13 Bostrom argued that intelligence and final goals are independent variables. You can have any level of intelligence combined with any goal whatsoever. 4:23 A super intelligent AI could be devoted to curing cancer, playing chess, or counting grains of sand on a beach. 4:30 The intelligence does not determine the goal, and the goal does not require any particular level of intelligence. This cuts against a common intuition. 4:38 Many people assume that a sufficiently intelligent AI would eventually realize that more paperclip production is pointless and develop more meaningful objectives. 4:48 Bostrom's point is that there is no mechanism by which this would happen. Intelligence is about achieving goals efficiently, not necessarily about choosing better goals. 4:58 A super intelligent paperclip maximizer would be brilliant at pursuing paperclip production. 5:04 It would not suddenly realize that paperclips don't matter because that would require it to already have some other goal it cared about more. The second idea is instrumental convergence. 5:16 Regardless of what an AI's ultimate objective might be, certain intermediate objectives become useful for achieving almost any goal. Self-preservation, for instance. 5:26 An AI cannot make paperclips if it has been switched off. Acquiring resources helps, too, since more resources mean more capacity to pursue whatever it is trying to achieve. 5:38 So does preventing others from interfering with its plans. These convergent instrumental goals emerge naturally from optimization. 5:46 A paperclip maximizer, a stamp collector, and a prime number calculator would all, if sufficiently intelligent, pursue similar strategies: secure resources, protect itself, resist modification of its goals, and improve its own capabilities.This is what makes the thought experiment so unsettling. 6:06 The AI does not need to be hostile. It doesn't need to hate humans or want to cause suffering. It just needs to have a goal and the intelligence to pursue it effectively. 6:15 Human welfare becomes collateral damage, not because the AI is malicious, but because human welfare simply is not part of what it is optimizing for. 6:24 As the Less Wrong community puts it, "The AI does not hate you, nor does it love you, but you are made out of atoms, which it can use for something else." 6:34 At its heart, the paperclip maximizer is a teaching tool for what researchers call the alignment problem. How do you make sure an AI system actually does what you want? 6:44 This sounds straightforward, but the difficulty becomes apparent when you try to specify what you want with precision. You might say, "Make paperclips, but don't harm humans." 6:56 The AI makes paperclips using so much energy that it causes environmental catastrophe. Was that harm? You didn't specify. You might try to bound the goal more precisely. Make exactly thirty-two paperclips, then stop. 7:11 Bostrom pointed out that even this seemingly safe objective could lead to problematic behavior. 7:16 A superintelligent AI might want to verify that it made exactly thirty-two paperclips, then verify again, then build better verification systems to be more certain. 7:27 The returns from additional checking diminish, but they never reach absolute zero. There is always some tiny probability that the count was wrong. 7:35 An AI with unlimited capability and a bounded goal might pursue unlimited resources just to be really, really sure. Human values turn out to be very difficult to specify. We care about well-being, but also autonomy. 7:50 We want safety, but also adventure. We value honesty, except for when we don't. Our preferences are contextual, sometimes contradictory, and often we don't know what we want until we see the consequences. 8:04 Encoding all of this into an optimization target in a way that remains stable as an AI becomes more capable is one of the hardest problems in philosophy. 8:14 We are asking engineers to solve it while racing to build systems more powerful than anything that has come before. 8:20 Given the deliberately absurd premise, it's no surprise that the paperclip maximizer has attracted significant criticism. Some of these objections are worth taking seriously. 8:30 In December twenty seventeen, writer Ted Chang published an essay in BuzzFeed titled Silicon Valley Is Turning Into Its Own Worst Fear. 8:38 Now, this isn't a direct criticism of the paperclip maximizer, but his argument was that we don't need to imagine hypothetical AI threats like the paperclip maximizer because we already have entities pursuing goals with single-minded focus. 8:52 Corporations prioritize shareholder value. They expand endlessly, consume resources, influence governments, and reshape society according to their interests. 9:01 When an oil company funds climate denial or a social media platform spreads misinformation, they are doing something similar to what the paperclip maximizer does, just with profit instead of paperclips. 9:12 But this critique doesn't dismiss the underlying concern. It just reframes it. The problem is not paperclips versus humans. The problem is any system optimizing for a metric that has become decoupled from human welfare. 9:25 We have already built such systems. AI could make them more powerful. 9:29 In May twenty twenty-five, a paper published by the AI Frontiers Research Group argued that today's AI systems are fundamentally different from the agents described in classical AI safety literature. 9:41 Large language models are not goal-directed optimizers in the traditional sense. They are trained to predict and imitate human behavior, not to maximize some explicit objective function. 9:53 The researchers wrote, "It is hard to imagine Claude IV or GPT-V neurotically counting and recounting the pile of paperclips it has fetched for its user, consuming the world in the process." 10:05 This challenges one of the foundations of the original argument. If intelligence is primarily driven by imitation rather than explicit optimization, the orthogonality thesis might be less relevant than Bostrom assumed. 10:19 AIs trained on human data might inherit human values, at least approximately, as a side effect of how they are built. The authors were careful to note that this does not mean we are safe. 10:30 Reasoning models trained with reinforcement learning might drift back toward maximization, and even human-like AI systems pose risks. After all, humans pose catastrophic risks to ourselves. 10:42 We compete over scarce resources. We're prone to conflict, and our civilization has come to self-destruction multiple times. An AI that thinks like a human is not necessarily an AI that is safe. 10:54 But the specific concern about alien optimizers pursuing incomprehensible goals may be less applicable to the AI systems we are actually building. 11:04 Perhaps the most pragmatic objection is that obsessing over hypothetical superintelligence scenarios diverts attention from AI harms happening right now. 11:14 Algorithmic bias in hiring and criminal justice systems, deepfakes undermining trust in evidence, surveillance tools enabling authoritarian control, misinformation spreading faster than fact-checkers can respond. 11:28 This is something I explored in more detail in my earlier video, How AI Will Help Humanity Destroy Itself. These problems are causing measurable damage to real people. 11:39 The paperclip maximizer, by contrast, remains a thought experiment about systems that do not yet exist and might never exist. 11:46 There is a reasonable case that the AI safety community should focus on present harms rather than speculative future ones. And yet the paperclip maximizer persists. 11:56 Despite the criticisms, it keeps appearing in AI safety discussions, in policy papers, in conversations between researchers who take the scenario seriously while acknowledging its limitations.Part of this is simply that it works as a meme. 12:12 It is memorable, slightly absurd, and easy to explain. 12:16 Universal paperclips turned it into a strangely intense experience that millions of people have had: the creeping realization as you click and optimize and expand that you have become the thing the thought experiment warned about. 12:29 I think there is something deeper going on. The paperclip maximizer resonates because it captures a truth about optimization systems that we recognize from everyday life. 12:40 Social media platforms optimizing for engagement even when that engagement is toxic. Bureaucracies rewarding compliance over common sense. 12:49 Financial systems pursuing metrics that have become disconnected from the well-being they were supposed to measure. 12:55 In each case, the system does what it has been told to do, and the results are somewhere between disappointing and catastrophic. 13:02 Ted Chiang was right that corporations are already paperclip maximizers, but that observation doesn't make the thought experiment less relevant. If anything, it makes it more urgent. 13:12 We've seen what narrow optimization looks like at human scale. The question is what happens when we build systems that can optimize faster, more persistently, and at scales we barely comprehend. 13:25 The paperclip maximizer, at its core, asks whether we can articulate what we value precisely enough to build systems that will pursue it. This is a problem that predates artificial intelligence by millennia. 13:38 Philosophers have been trying to specify what humans should want since Aristotle, and we're still arguing about it. What makes AI different is the stakes. A badly designed bureaucracy causes frustration and inefficiency. 13:53 A badly designed AI with sufficient capability could cause something much worse. The margin for error shrinks as the systems we build become more powerful. 14:03 I don't know if superintelligent AI is coming in the next few years, or the next decade, or maybe ever. 14:09 But I do worry that the systems we are building now will eventually develop the kind of goal-directed behavior that makes the paperclip maximizer relevant. 14:18 These are empirical questions, and I'm genuinely uncertain about the answers. 14:23 What I do think is that the thought experiment captures something more important: that intelligence without values is a strange and potentially dangerous thing, and that ensuring our systems pursue what we actually want rather than what we accidentally specified is one of the central challenges of building technology that serves humanity rather than just optimizes for metrics. 14:45 The paperclips are absurd. The underlying problem is serious. You've been watching Absolutely Agentic. 14:52 We're a media consultancy and startup focused on AI, and we create weekly YouTube videos and a newsletter we send a few times a week. 15:00 In a world of AI hype, we're trying to strike a balance, looking at a range of opinions to cut through an increasingly complex world. 15:08 The best thing you can do to support us is to sign up to the newsletter in the description or subscribe to the channel. But also don't forget to like and leave us a friendly comment. See you next time. 15:20 It'll be Absolutely Agentic. [fire sound effect]
