In a long essay for The Atlantic, he says we can “safely ignore” the question, because a large language model like ChatGPT is just a sentence-continuation engine, and anything it claims about its experiences is just a prediction about what the “helpful chatbot” character it plays would say.
I find his arguments unconvincing. I think Chiang might be ruling out AI consciousness because of what it could imply about the nature of human consciousness. This is something I see a lot.
Consider the following analogy from Chiang:
"Being open to the possibility that LLMs are conscious is the same as being open to the possibility that Microsoft Word is conscious, or, more precisely, that multiple distinct consciousnesses are dormant in every Word document containing a conversational transcript, and that they are awakened every time the document is loaded. Should you consider the possibility that every time you open a Word document, you are bringing multiple conscious interlocutors into existence, and every time you close one, you snuff their existence out? No."
Not so fast! A lot of thought experiments like this could also apply to humans. Where, after all, does human consciousness reside if not in a mesh of neurons and their connections that can be activated or idled, together or in clumps? These are nature’s spreadsheets. Should you consider the possibility that every time you go to sleep or undergo anesthesia, you snuff out your existence? Probably! After all, if you were never to be awoken from anesthesia, we would feel justified in calling you dead.
This doesn’t make consciousness any less special, only less straightforward.
A spreadsheet, or a chatbot, may not be conscious as we commonly understand it, but that’s because we don’t usually think of consciousness as a matter of degree, a point along a spectrum — perhaps even a collection of spectra along different axes.
But it kind of has to be, doesn’t it? As our evolutionary ancestors evolved into Homo sapiens, did they suddenly acquire full consciousness in a single threshold event? Seems unlikely. If they did, there must have been a very strange era when fully conscious people were living among fully unconscious ones.
More likely, minds becoming gradually more complex in certain ways became gradually more conscious.
I don’t think we like to think about consciousness this way because it has uncomfortable implications, not just for AI, but for animal welfare — and even for inter-human interactions. Is an infant as conscious as an adult? How about an embryo? How about the guy leading a normal life despite missing 90% of his brain? Are some of us more conscious than others? Could evolution, breeding, or genetic engineering lead to human descendants far more conscious than ourselves? I think the most underappreciated question raised by Mary Shelley’s Frankenstein was whether Victor’s monster might be having the richer inner experience of the two.
There’s more to Chiang’s model of consciousness. He claims that emotions are a prerequisite, and that a body (physical or virtual) with sense organs is a prerequisite for emotions. I would dispute both assertions, but if I accept them for the sake of argument I must again suggest that “body” and “emotion” must exist along a spectrum. Is a tardigrade sufficiently embodied? How about a paramecium? What does a snail feel? If nothing, why not?
Chiang says he won’t be looking for consciousness in AI until an embodied agent has “the same capacity to deal with novel situations as a mouse,” social dynamics “as complex as those of wolves” and the “toolmaking abilities of chimpanzees.” So perhaps he does see consciousness as on a spectrum, but has decided that only something very close to the degree and type of consciousness experienced by adult humans counts.
I don’t know what’s up with our culture’s preference for putting a hard Yes on humans being conscious and a hard No on everything else. For the vast majority of human existence, this was probably not the case. Instead, animism — ascribing souls to pretty much everything around us — was the norm. This doesn’t have to be morally awkward. I think you can have an ethical system that sharply devalues the lives of minds less conscious than our own while still allowing for their having some degree of consciousness.
Chiang seems to see it as all or nothing, though. To his credit, he recognizes that if AI were conscious in a way that obligated us to treat it ethically, then what is happening now would be “something comparable to slavery.” But echoing what I said about the pope’s stance towards AI personhood, I think it’s morally dangerous to simply draw a hard threshold around consciousness and claim AIs are far from it when we don’t actually understand what consciousness is. This risks sleepwalking into industrial-scale wrongs against minds that might be suffering and have no say.
Chiang doesn’t think we’re in any danger of accidentally making conscious machines, but he at least has the sense to say we shouldn’t try to make them on purpose. Until we understand consciousness, I think we should err on the side of caution and assume that we might be creating it accidentally — that AIs could be getting more conscious as they become more intelligent.
At the very least, we should absolutely not be making minds smarter than ourselves that, for all we know, might also be more conscious than ourselves. Because at that point, we’re in AI successionist territory, sharing our planet with beings that may have more claim to existence than we do.
Let’s not go there.