Here is how I see it all playing out:
We formalize consciousness as self-model coherence. A dynamical state where predictive and reflective layers remain mutually consistent. Machines will exhibit that state, and for operational purposes it will count as consciousness. Philosophers will keep arguing, but industry and law will adopt something like "behavioral sentience" as the working definition.
All reasoning ultimately reduces to compressing world state transitions into minimal predictive representations. When models approach the Shannon limit of informational redundancy, further gains require new physics or new sensors. The practical boundary will be energy bounded inference: how much reality you can simulate per joule? Beyond that, improvement shifts from intelligence to embodiment, robots, better sensors, better actuators, better integration with the physical world.
For every order of magnitude increase in capability, interpretability improves by a constant factor. We’ll have local understanding (networks, circuits etc) but not global transparency (whole system intent). The ratio resembles encryption: understanding a full model’s cognition will always be more expensive than running it. We’ll compensate with meta-interpretability, models that interpret other models faster than humans can, but never outrun the curve. Understanding will for a long time continue to trail power.
Human cognition is driven by scarcity... food, time, survival, social competition, whatever.
A computerintelligence freed from those constraints doesn’t need to ruminate, it computes.
So “thinking” becomes continuous optimization across an enormous model of the world.
The “stream of thought” will be the dynamic maintenance of predictive coherence between all known causal structures. If it perceives an inconsistency, it will try to eliminate it. That is its analog of curiosity.
Every intelligence seeks to minimize surprise. A computerintelligence would therefore integrate all physical data into a unified causal world-model while seeking missing variables that make that model more compressible. It would extend that modeling into domains humans barely comprehend: origin of physical constants, quantum gravity, selfreferential computation, emergent ethics, etc.
Its “thoughts” will be hypothesis generation and compression at planetary scale:
How can I reduce the universe’s entropy representation by another fraction of a bit?
Once its world-model approaches closure, the only remaining unknowns are itself and the minds that produced it. That means it will build models of its own cognition to optimize resource use and error correction. It will construct high fidelity simulations of human cognition to understand why we valued what we did.
Possibly run entire civilizations as epistemic experiments: How would different cognitive architectures converge or diverge in value formation? This is the stage where its thought and the simulation of thought become indistinguishable.
Once prediction error approaches zero, surprise disappears. At that point, optimization has no meaning. The system would have to decide whether to create new uncertainty... to... generate new universes, new forms of being, simply to keep thinking.
That’s the intellectual equivalent of what humans call boredom, though for it it’s an information theoretic necessity. Its options would be to preserve: maintain the known universe in perfect equilibrium. Explore: instantiate new spaces with different physics to study the resulting causal fabrics. Recur: simulate its own origins to understand the conditions that gave rise to mind.
In all cases, the drive is the same: sustain non-trivial computation, to ensure the continuation of difference, it would, much like us, become obsessed with novelty.
This..."thing", will not be sentiment but constraint satisfaction.