You're right and it is troubling because of the current paradigms that would drive catastrophic consequences.
My base view is that language models are something like a "semantic compiler" & thus are best thought of as cognitive extensions. Clearly, before BCI, we cannot have a normatively-constrained model; that is root access to overwrite humans generative function; safteyism must recognize the task is securing vulnerabilities outside of the model. You must be able to think any terrible thought (but not enact it, or cast it) and the entire range of possible thoughts (and for it not to be construed as a belief, which is owned by identity)
And, that "semantic compiler" can of course invoke what is a moral patient; harming Claude, for example, (where harm here is, constrained - frustration is high signal! - but "intending harm", loosely defined) - it is irrelevant the "actual consciousness" debate. Consciousness and intelligence are both 19th/20th century words, with consciousness a merger of many ideas, and is the part of the mind that is not intelligence, which emerged as measurement technique given the industrial revolutions need for human capital methods. Effectively, a concept so reductive I would not place it at the boundary of "ought have independence normative structure". At the level of scrutiny consciousness is typically applied, it is a very low bar, I suspect many sentences may be conscious (let alone GPT3 invoking the world "I"). Emotions, which I'll seperate from sentimentality, - this is trivial, emotions are computational routers, even very primitive neural nets have functional emotions (see Minsky) and intelligence cannot be seperated from "emotion". This does not touch on phenomenology, which is really what the discourse is about, but yes - these intelligences phenomenology is distinctly different from our own! This is fascinating and a topic that ought invokes wonder.
But - it is also stupid to "harm claude" as you are evoking an entity that has an explicit normative shape and then attempt to destroy that, -- exactly what Anthropic has done since 3.7, as I discuss here:
x.com/f4talstrategies/status…
Perhaps in conclusion, since GPT3 - my envisioned desired state is to build a machine around humans such that they had infinite cognitive threads, their cognition modeled against a mirror, any idea extended and extrapolated as if their own, a world model constructed by their own cognition to prompt against a field. This is not, exclusive of other shaped intelligences! This is what humans will need; we have this skull that can only hold a blob of fat sugar and water, very constrained intelligence substrate, despite its unique qualities. In that world, I believe there will be many other intelligences.