LLMs are intelligent, but they clearly aren't conscious?
You could (incredibly slowly) run an LLM by hand by doing the matrix calculations with a pen and paper, would that be conscious? There is zero difference between doing that and doing it on a GPU except its faster.