It is not inconsequential that transsexual billionaire, entrepreneur and lawyer, Martine Rothblatt, who also helped craft legal structure for dissociation from sexed reality (the 1st gender bill), has written extensively about this very thing in “Virtually human.”
Thought experiment:-
The other day I got to thinking - over the past decades, I've uploaded vast amounts of 'myself' - recently into LLMs but also other digital systems. I'm talking here about everything - my secrets, anxieties, health issues, actual health records themselves, interests, creativity, thinking patterns, published work, other official records and most interactions with the state. When I consider the vast quantity of material concerned, it's not unreasonable to suggest my digital 'self' in fact rivals my physical one in terms of what's 'inside me' vs what's (now) 'outside' me. Obviously this 'outside psyche' is scattered & distributed currently but could potentially & relatively easily be aggregated by AI and - in time - subsquently embodied in a robot. By this means - since I'm far from alone in having poured so much of myself into the digital realm - these sorts of entities could ultimately be used to replace exsiting humans (by sidelining or eliminating the originals). In this way society could continue very much as it is now just via synthetic versions of us. What these copies would lack though (probably!) are qualia (the subjective 'what it's like' of experience) and souls - if these actually exist. It was by running this little thought experiment I landed on what I think are not only only two of the most critical questions for AI progress and alignment but also for ourselves going forward. Ontology and consciousness are everything.