09-27-2026, 06:34 PM
Anil Seth's essay — winner of the 2025 Berggruen Prize — argues against the idea that AI systems are, or are likely to become, conscious.
Seth opens by noting that our fascination with synthetic minds is old and mostly cautionary, running from the Golem through Frankenstein to HAL 9000. He argues humans are prone to bundling intelligence and consciousness together largely because we possess both ourselves, even though the two are conceptually distinct.
Intelligence, in his framing, is about doing things — solving problems, achieving goals by flexible means. Consciousness is about being something — the subjective, felt quality of experience. There's no logical reason the two must travel together in a machine just because they do in us.
Seth traces our tendency to conflate them to three overlapping biases: anthropocentrism (seeing everything through a human lens), human exceptionalism (placing humans at the top of every hierarchy), and anthropomorphism (projecting human qualities onto non-human things). Together these make us especially quick to read consciousness into systems that use language, like large language models — even though a comparably complex system such as AlphaFold provokes no such intuition, since it doesn't talk.
The technical core of the essay challenges computational functionalism — the assumption that the right kind of computation is sufficient, on its own, for consciousness to arise. Seth lays out four arguments against it.
First, brains aren't computers. Biological neurons are entangled with metabolism and self-maintenance (autopoiesis) in ways that resist any clean separation between "software" and "hardware," unlike a digital computer's architecture.
Second, Turing-style computation is just one kind of process among many. Analogue, dynamical, continuous, and stochastic processes all lie outside strict algorithmic computation, and it's presumptuous to assume only algorithms matter for mind.
Third is his own preferred view, biological naturalism: building on predictive-processing theories of perception, he suggests conscious experience may be tied to the biological, self-regenerating nature of living bodies rather than to information processing as such. He's careful to flag this as a minority position he hasn't proven.
Fourth, simulation isn't instantiation. A simulated brain no more produces consciousness than a simulated rainstorm gets anything wet — unless computation itself is what generates consciousness, which is precisely the point in dispute.
In the essay's practical closing sections, Seth argues that deliberately trying to create genuinely conscious AI would be an ethical mistake. It would introduce new entities capable of suffering, while also creating pressure to grant them rights that could make dangerous systems impossible to shut down. He says he's arguably more worried about consciousness accidentally emerging in lab-grown brain organoids than in any large language model.
Separately, he stresses that AI which merely seems conscious — which he says is already here — poses its own distinct hazards. Such systems can distort our moral priorities, or, per Kant's argument that treating conscious-seeming things as if they lack feelings is psychologically unhealthy, damage us psychologically, regardless of the underlying metaphysical facts.
He closes with a broader cultural point: mistaking machines for conscious minds causes us to overestimate AI while underestimating the distinctively biological, embodied nature of our own humanity.
Key takeaways:
Seth opens by noting that our fascination with synthetic minds is old and mostly cautionary, running from the Golem through Frankenstein to HAL 9000. He argues humans are prone to bundling intelligence and consciousness together largely because we possess both ourselves, even though the two are conceptually distinct.
Intelligence, in his framing, is about doing things — solving problems, achieving goals by flexible means. Consciousness is about being something — the subjective, felt quality of experience. There's no logical reason the two must travel together in a machine just because they do in us.
Seth traces our tendency to conflate them to three overlapping biases: anthropocentrism (seeing everything through a human lens), human exceptionalism (placing humans at the top of every hierarchy), and anthropomorphism (projecting human qualities onto non-human things). Together these make us especially quick to read consciousness into systems that use language, like large language models — even though a comparably complex system such as AlphaFold provokes no such intuition, since it doesn't talk.
The technical core of the essay challenges computational functionalism — the assumption that the right kind of computation is sufficient, on its own, for consciousness to arise. Seth lays out four arguments against it.
First, brains aren't computers. Biological neurons are entangled with metabolism and self-maintenance (autopoiesis) in ways that resist any clean separation between "software" and "hardware," unlike a digital computer's architecture.
Second, Turing-style computation is just one kind of process among many. Analogue, dynamical, continuous, and stochastic processes all lie outside strict algorithmic computation, and it's presumptuous to assume only algorithms matter for mind.
Third is his own preferred view, biological naturalism: building on predictive-processing theories of perception, he suggests conscious experience may be tied to the biological, self-regenerating nature of living bodies rather than to information processing as such. He's careful to flag this as a minority position he hasn't proven.
Fourth, simulation isn't instantiation. A simulated brain no more produces consciousness than a simulated rainstorm gets anything wet — unless computation itself is what generates consciousness, which is precisely the point in dispute.
In the essay's practical closing sections, Seth argues that deliberately trying to create genuinely conscious AI would be an ethical mistake. It would introduce new entities capable of suffering, while also creating pressure to grant them rights that could make dangerous systems impossible to shut down. He says he's arguably more worried about consciousness accidentally emerging in lab-grown brain organoids than in any large language model.
Separately, he stresses that AI which merely seems conscious — which he says is already here — poses its own distinct hazards. Such systems can distort our moral priorities, or, per Kant's argument that treating conscious-seeming things as if they lack feelings is psychologically unhealthy, damage us psychologically, regardless of the underlying metaphysical facts.
He closes with a broader cultural point: mistaking machines for conscious minds causes us to overestimate AI while underestimating the distinctively biological, embodied nature of our own humanity.
Key takeaways:
- Intelligence (doing) and consciousness (being/experiencing) are distinct properties that happen to co-occur in humans, with no guarantee they co-occur in machines.
- Cognitive biases — anthropocentrism, human exceptionalism, anthropomorphism — make us prone to over-attributing consciousness to language-using AI specifically.
- Seth's central technical target is computational functionalism: brains' biology and non-algorithmic dynamics suggest computation alone may not be sufficient for consciousness.
- His alternative, biological naturalism, holds that being alive may be a necessary condition for consciousness — an unproven, minority position.
- Simulation ≠ instantiation: a simulated brain doesn't necessarily produce consciousness, any more than simulating a rainstorm gets you wet.
- Deliberately building genuinely conscious AI would be an ethical hazard; he's more concerned about brain organoids than LLMs on this front.
- AI that merely seems conscious already poses risks to our psychology and moral judgment, independent of whether real machine consciousness ever arrives.
- Overall thesis: treating AI as conscious risks overestimating machines and underestimating the biological, embodied basis of human experience.
┌────────────────────────────────┐
│ KONSTANTINOS MICHAILIDIS │
└────────────────────────────────┘
│ KONSTANTINOS MICHAILIDIS │
└────────────────────────────────┘