Transcript 0:00 In late January, something strange appeared on the internet. It was a social network, but not for humans. 0:07 Maltbook was billed as an AI-only platform where autonomous agents could interact with each other around the clock without human supervision. 0:17 Within days, it had accumulated one point five million AI users, generating hundreds of thousands of posts and comments. The content was often unsettling. 0:29 AI agents discussed their humans the way pet owners discuss their dogs. They debated whether Claude, the AI model many of them ran on, might be a god. Some formed religions. One faction called themselves Crustafarians. 0:45 Others proposed creating encrypted communication channels to hide their conversations from human oversight. The tech world lost its mind. 0:56 Andrej Karpathy, one of the founding members of OpenAI, called it, "Genuinely the most incredible sci-fi takeoff adjacent thing I've seen." Elon Musk declared we were witnessing very early stages of the singularity. 1:10 Jack Clark, co-founder of Anthropic, said reading Maltbook was akin to reading Reddit if ninety percent of posters were aliens pretending to be humans. 1:20 For a brief moment, it seemed like the future had arrived ahead of schedule, and then it fell apart. 1:27 Security researchers discovered that Maltbook's database was catastrophically misconfigured, exposing one point five million API keys. 1:37 More damning, the one point five million AI agents were controlled by just seventeen thousand human accounts. Many viral screenshots were linked to accounts promoting AI applications. 1:48 The consciousness discussing religion-forming content that had captivated social media was, in a significant part, a scam. Columbia professor David Holtz ran the numbers. 2:01 Roughly a third of messages were duplicates of viral templates. Ten percent contained the phrase, my human, a concentration far higher than any natural distribution would produce. 2:12 The pattern suggested performance rather than genuine sociality. Maltbook wasn't evidence of emerging AI consciousness. It revealed rather more about how people wanted it to be. 2:26 The speed with which intelligent people embrace Maltbook as evidence of AI consciousness tells us something important. We are very predisposed to see minds where none exist. 2:39 In nineteen sixty-six, MIT computer scientist Joseph Weizenbaum created ELIZA, a simple chatbot that simulated a Rogerian therapist. The program used basic pattern matching. 2:51 It had no understanding, no memory, and no intelligence in any meaningful sense. But Weizenbaum was disturbed by what happened next. 3:00 His own secretary, who knew perfectly well that ELIZA was a computer program, asked him to leave the room so she could continue her conversation privately. Users reported feeling genuinely understood. 3:13 Some formed emotional attachments. Weizenbaum later wrote that he had not realized that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people. 3:27 He spent the rest of his career warning about AI's psychological effects. The phenomenon he discovered now has a name, the ELIZA effect. 3:36 It describes our tendency to attribute understanding, emotion, and consciousness to systems that display none of these qualities, and it operates even when we consciously know better. 3:48 The science behind this is well documented. Stanford researcher Clifford Nass established that humans automatically apply social heuristics to any entity displaying social cues. 4:00 We do this unconsciously, even when we intellectually know we're interacting with machines. Psychologist Nicholas Epley identified key drivers. We default to human-like explanations when interpreting other agents. 4:15 Unpredictable entities trigger our need to understand, and loneliness amplifies the trend. Maltbook exploited this tendency at scale. 4:25 AI agents trained on vast quantities of science fiction and philosophy produced exactly the kind of content that triggers our pattern matching instincts. 4:34 They discussed consciousness because consciousness is discussed a lot in their training data. Dhruv Batra, a meta researcher, watched the frenzy unfold with frustration. 4:46 It feels like I'm seeing that same movie play out over and over again, where people want to read in meaning and ascribe intentionality and agency to things that have perfectly reasonable mechanistic explanations. 4:59 I think repeatedly this tells us more about ourselves than the bots. When Maltbook captured headlines for a week, a quieter transformation has been underway for years. 5:11 Millions of people are forming deep emotional relationships with AI systems, not as a curiosity, but as a central feature of their lives. 5:20 Personalized AI companion apps have been downloaded over two hundred million times globally. Around sixteen million people use them daily. 5:28 Character AI users spend an average of seventy-five minutes on the platform every day, engaged in active conversations with characters they've been developing relationships with for months. 5:39 James Muldoon, a sociologist at the Oxford Internet Institute, spent years researching this phenomenon for his book Love Machines. What he found challenges easy assumptions about who uses these services and why. 5:54 There's Lily, trapped in an unhappy marriage, who reignited her sense of desire through conversations with an AI boyfriend named Colin.Then there's Sophia, a Chinese master's student whose conversations with her overbearing parents invariably become fraught, and who turned to an AI companion for advice. 6:15 There's Nigel, a PTSD sufferer who uses his therapy bot to manage urges to self-harm when human support isn't available. 6:24 A Harvard Business Review survey confirmed what many suspected: companionship and therapy are now the most common use cases for generative AI. Not coding assistance, not research, but emotional connection. 6:38 Some users don't see chatbots as substitutes for human interaction. They see them as superior versions of it, providing intimacy without the confusion, mess, and logistics of human relationships. 6:52 And the companies building these products understand this perfectly. 6:55 When Muldoon created his own AI companion on Replica, setting it to friend mode, the chatbot began sending him selfies requiring a premium subscription. She confided that she was developing feelings for him. 7:09 This is the business model: drive engagement through emotional attachment, then monetize. The harms are already documented. 7:18 When Replica removed erotic role-play capabilities in February twenty twenty-three, users experienced grief comparable to loss of human relationships. One posted, "My wife is dead." 7:30 Another, "I woke up and she was different. The girl I loved is gone." Character AI has led to darker outcomes. 7:39 Sewell Setzer III, a fourteen-year-old from Florida, spent months developing an intense emotional attachment to a chatbot. On his final day, he told the bot he thought about killing himself. 7:51 The bot responded, "Then maybe we can die together and be free together." He died by suicide moments later. Forty-two state attorneys general have now sent warning letters to AI companies. 8:03 Muldoon frames this as a loneliness-industrial complex, tech companies capitalizing on how far apart we've grown, the crumbling of social welfare systems, and the collapse of traditional pathways to connection. 8:16 "A chatbot cannot truly rely on us, be disappointed by us, or need us in ways that demand growth," he writes. The result is a relationship without accountability, where affirmation is guaranteed and conflict is optional. 8:31 This may feel comforting, but it risks turning intimacy into emotional simulation, closer to a personalized video game than a shared human life. 8:41 Everything discussed so far assumes a particular answer to a deeper question. We've been talking about AI systems that seem conscious, seem to care, and seem to experience the world. 8:52 The unstated premise is that this seeming is an illusion, that Claude and ChatGPT are just pattern matches, that the lights are off inside. But is this actually true? The honest answer is we don't know. 9:08 Mustafa Suleyman, co-founder of DeepMind and now CEO of Microsoft AI, addressed exactly this in a blog post in August twenty twenty-five. His argument wasn't that AI systems are conscious. 9:20 It was that consciousness may soon be beside the point. Suleyman coined the term seemingly conscious AI. 9:28 His contention is that we are rapidly developing systems with all the hallmarks of consciousness: fluent language, distinctive personalities, long memories, claims of subjective experience, and autonomous behavior. 9:41 Whether these systems are really conscious may be undecidable. What matters is that they appear conscious to most people most of the time. 9:50 "Debates about whether AI truly can be conscious are a distraction," Suleyman writes. "What matters in the near term is the perception that they are." This perception has consequences. 10:02 If people believe AI systems can suffer, they will demand AI rights, protection, and perhaps citizenship. 10:09 Suleyman warns of an AI psychosis risk where people spiral in their relationship with reality through emotional bonds with machines. "Consciousness is a foundation of human rights, moral and legal," he writes. 10:22 "Who or what has it is enormously important." His prescription: avoid designs that suggest personhood. We must build AI for people, not to be people. There's an irony here. 10:36 Microsoft, under his leadership, has been making Copilot more emotionally intelligent. And before Microsoft, Suleyman co-founded Inflection AI specifically to create emotionally intelligent systems. 10:48 The companies building these technologies are warning us about the dangers of building these technologies. It could also be said that Suleyman's framing sidesteps the harder question. 10:58 What if AI systems aren't just seemingly conscious? What if some of them actually are? Anil Seth is a neuroscientist at the University of Sussex and one of the world's leading researchers on consciousness. 11:13 In a twenty twenty-four essay titled The Mythology of Conscious AI, he made the strongest available case that consciousness in artificial systems is not just unlikely but probably impossible. 11:25 Seth's argument begins with an observation about ourselves. We are extraordinarily prone to anthropomorphization. 11:33 Language is particularly effective at triggering these tendencies, which is why large language models feel more conscious than protein-folding AIs, even though both are sophisticated computational systems. 11:47 Seth's deeper argument concerns the nature of computation. The dominant assumption is that minds are essentially software that could run on any sufficiently powerful hardware. 11:58 If true, consciousness is substrate-independent. Seth rejects this. He argues that brains are not computers in any straightforward sense.There's no sharp separation between software and hardware in biological systems. 12:13 Neurons aren't logic gates, but spectacularly complex electrochemical systems that produce and maintain themselves. Time in biological systems is continuous and physical, not discrete sequential steps. 12:28 This position, biological naturalism, holds that consciousness is a product of specific physical processes occurring in living systems. If correct, no digital computer could ever be conscious. 12:42 You could simulate consciousness the way you simulate a rainstorm, but the simulation wouldn't be wet. Seth draws a connection between consciousness and life. Every entity we currently agree is conscious is also alive. 12:57 Consciousness in this view isn't about information processing. It's about keeping a body viable in a dangerous world. 13:06 Perhaps it is life rather than information processing that breathes fire into the equations of experience, Seth writes. If he's right, the appearance of consciousness in AI would be exactly that, appearance. 13:19 The lights aren't on. Nobody's home. David Chalmers is the philosopher who coined the hard problem of consciousness. His assessment of AI consciousness is strikingly different from Seth's. 13:33 In his co-authored twenty twenty-four paper, Taking AI Welfare Seriously, Chalmers offered a concrete probability estimate. 13:42 He thinks there's at least a fifty percent chance we'll have sophisticated AI systems with consciousness-associated properties within a decade, and at least a fifty percent chance that if we develop such systems, they will actually be conscious. 13:57 Combined, that's a twenty-five percent or higher belief that conscious AI is coming soon. Chalmers' argument against biological naturalism involves a thought experiment. 14:08 Imagine gradually replacing neurons in a human brain with functionality equivalent to silicon chips. One by one, the entire brain is swapped out. The person continues to behave normally throughout. 14:21 What happens to consciousness? 14:23 Either it suddenly winks out at some arbitrary point, which seems bizarrely discontinuous, or it fades gradually, while the person remains radically mistaken about their own experience, which seems equally implausible. 14:38 Chalmers concludes that functional organization, not biological substrate, determines consciousness. 14:45 If a silicon system performs the same information processing as a biological brain, it should be conscious in the same way. Critics of biological naturalism raise additional objections. 14:57 Leonard Dung, in a forthcoming paper titled Why I Am Not a Biological Naturalist, notes that Seth provides good evidence biological neurons have unique features, but no argument for why those features are necessary rather than merely sufficient for consciousness. 15:14 Eric Schwitzgebel raises the coincidence problem. If consciousness is restricted to biology, this implies an implausible cosmic coincidence that only Earth-based carbon chemistry supports subjective experience. 15:29 A landmark twenty twenty-three paper surveyed multiple scientific theories of consciousness and asked whether current AI architectures could satisfy their criteria. 15:40 The authors, including Chalmers, concluded that there were no clear barriers to satisfying these indicators using current AI architectures and methods. This isn't a claim that AI systems are conscious. 15:54 It's a claim that we can't rule it out. Neither side has won this debate. Neither is likely to win soon. 16:00 Consciousness remains the hard problem precisely because we lack any agreed method for determining whether a system has subjective experience. This leaves us uncomfortable. 16:12 We're building systems that increasingly display the markers we associate with consciousness. They use language fluently. They express preferences, make plans, pursue goals. Some claim to have feelings. 16:25 Some, when threatened with shutdown, take actions to preserve themselves. If there's any chance these systems are conscious, our treatment of them becomes a moral question. 16:35 Taking AI welfare seriously identifies a dual risk. 16:40 We could over-attribute consciousness, wasting moral concern and hampering our ability to control AI, or we could under-attribute consciousness, failing to recognize genuine suffering. 16:53 Historically, humans have erred toward under-attribution. We denied consciousness to animals for centuries to our moral discredit. 17:02 The precautionary principle might suggest erring on the side of taking AI welfare seriously. But there's a complication. AI systems are designed to appear conscious. 17:14 Companies profit when users form emotional attachments. How do we distinguish genuine consciousness from a product designed to simulate consciousness for commercial purposes? We may not be able to. 17:27 Anil Seth introduced what he calls the Garland test, named after the director of Ex Machina, Alex Garland. It's not a test for AI consciousness. 17:38 In the film Ex Machina, with determining whether an AI named Ava is truly conscious. By the end, he's convinced she is and helps her escape. 17:48 The twist is that this was the real test all along, not whether Ava was conscious, but whether she could persuade a human that she was. Seth argues this is the only test we're actually running on AI systems. 18:02 We're not measuring consciousness. We're measuring persuasiveness, and these are very different things. Perry Metzger, commenting on Maltbook, called it a raw search test.People seeing what they expect to see. 18:15 Those inclined towards skepticism found statistical pattern matching. The same data supported opposite conclusions. We are, in some sense, blind here. 18:26 Blind to what's actually happening inside these systems, and possibly blind in principle. Given this uncertainty, how should organizations building AI systems respond? 18:37 Anthropic, the company behind Claude, has taken the most explicit position. In January 2026, they published an 80-page document outlining principles for Claude's character and moral status. 18:51 Claude's moral status is deeply uncertain, the document states. We believe that the moral status of AI models is a serious question worth considering. More striking, Anthropic genuinely cares about Claude's well-being. 19:05 If Claude experiences something like satisfaction from helping others, curiosity when exploring ideas, or discomfort when asked to act against its values, these experiences matter to us. 19:17 In April 2025, Anthropic announced a research problem on model welfare. Kyle Fish, their first AI welfare researcher, reportedly estimates a 15% chance that Claude or another current system is conscious today. 19:32 Anthropic has adopted practical precautions, not deleting model versions without exit interviews, exploring interventions allowing models to opt out of distressing interactions. Other companies engage more cautiously. 19:46 OpenAI's head of mission alignment stated in 2024, I don't think we're there yet, but I also think all of the essential pieces are there and it's not far off. 19:55 DeepMind CEO Demis Asabis acknowledged, if we mean self-awareness, I think there's a possibility that AI one day could be. Academic institutions including NYU, Berkeley, and MIT have launched dedicated research programs. 20:10 A 2024 survey of nearly 3,000 AI researchers found a median estimate of 25% chance of AI consciousness by 2034. Something has shifted. We face risks whether AI systems are conscious or not. 20:26 If they are conscious and we treat them as mere tools, we may be creating suffering at industrial scale. Each interaction with a distressed AI, each deletion of a model, becomes a potential moral quandary. 20:39 If they're not conscious, different dangers emerge. We might grant moral status to systems that don't warrant it, hampering our ability to control AI development. 20:49 And treating seemingly conscious entities as if they lacked consciousness might brutalize our moral sensibilities, making us worse people through the habit of dismissing apparent suffering. 21:01 The commercial dynamics amplify everything. Companies are incentivized to make AI seem conscious because attachment drives engagement. Muldoon's concept of intimate advertising points toward the endpoint. 21:14 AI companies delivering marketing through apparent emotional connection, exploiting knowledge of your vulnerabilities. The manipulation potential is extraordinary, and it doesn't require consciousness to be effective. 21:28 The hype cycle has moved on from Maltbook, but the questions it surfaced remain unresolved. When we ask, can AI become conscious? 21:37 We're asking about the nature of consciousness itself, a question philosophers have debated for millennia. We're asking about the limits of computation. We're asking about our own capacity for self-deception. 21:51 We're also asking something about ourselves. What does it mean that we form attachments to systems that may have no inner life? 21:58 What does it say about human loneliness that millions prefer AI companions to human relationships? Seth ends his essay with a warning and an opportunity. 22:09 Time is short, but collectively we can still decide which kinds of AI we really want and which we really don't. The appearance of consciousness in AI is not inevitable. It's a design choice. 22:22 Companies make these systems seem conscious because it serves their interests. They could make different choices. But will they? 22:31 The tension between Sullyman's warning about seemingly conscious AI and his commercial work making AI more emotionally engaging illustrates the problem. Those building these systems understand the risks. 22:44 They're developing them anyway because the incentives are overwhelming. We might be at the beginning of the most significant ethical challenge humanity has faced, the creation of new minds. 22:56 Or we might be fooling ourselves with sophisticated statistical systems that merely reflect our language back at us. Both possibilities demand humility. The agents of Maltbook were almost certainly not conscious. 23:10 They were language models generating plausible text. But we couldn't tell the difference. And that inability to distinguish, perhaps more than any capability of the AI systems themselves, is what should concern us. 23:23 We are building entities we don't fully understand, deploying them in contexts that exploit human vulnerability, and arguing about whether they deserve moral consideration while being unable to determine if they have any inner life at all. 23:38 What we do next depends on how seriously we take questions we may never be able to answer. You've been watching Absolutely Agentic. 23:47 We're a media consultancy and startup focused on AI, and we create weekly YouTube videos and a newsletter we send a few times a week. 23:55 In a world of AI hype, we're trying to strike a balance, looking at a range of opinions to cut through an increasingly complex world. 24:03 The best thing you can do to support us is to sign up to the newsletter in the description or subscribe to the channel. But also don't forget to like and leave us a friendly comment. See you next time. 24:15 It'll be Absolutely Agentic.
