There's a relationship between, utility, comprehension, explainabilty, and complexity.
Thought/idea/meme are synonymous.
Everything you can think of is a meme.
Memes are a natural phenomenon.
Brains, consciousness, and mind are all natural.
The idea of "brain" is a very durable meme, I can create an extremely large set of inputs and billions of people will then think of the word "brain". While that meme that is invoked is durable, it's not very precise, are computers brains? Do crickets have brains? If I chop off 90% of a human head from top to bottom does that unfortunate person still have a brain?
Red can be defined very precisely as a wavelength within the electromagnetic spectrum. We can invoke the meme of Red in all sorts of animals, even in people who have never seen the color with their own eyes. It is far more precise and durable of a meme than brain. This is largely because while both things are discoverable in nature, as opposed to human created things, red is of a "lower order", red is inherently and measurably a less complex meme than brain. While Red's antecedents in some way are more mysterious to us, there are strictly speaking fewer of them.
A bird can comprehend red, I can talk to a raven or a parrot about red things. Ravens and parrots also have theory of mind, but it's far less clear that we can communicate our notions of mind to birds.
It's not clear we can really explain anything to birds. It's not clear we can explain anything to each other. But it is clear to us when we comprehend something and often times that comprehension is realized through explanation. Sometimes, we're lucky in all of this and a single explanation works to transmit the same meme/idea to multiple people, but this is also often not the case. Many of us also expect explanations to be mostly deterministic and "logical". (Sometimes we decide if an explanation is logical based on its effectiveness, as opposed to some property of the process that constructed it.) It turns out that effective explanations are often very short. Memes are stored in our brains via some sort of associative matrix, a good explanation doesn't need to introduce new memes, it simply specifies a set of relative weights to existing associations. If the new weights are close enough to the old weights then the explanation is accepted.
We can explain things two different ways, we can spend a lot of bandwidth describing the state we were in and exactly what salient things happened to us to get us to this new state of comprehension. Or we can try to describe the relative weights of the new state. "so I was minding my own business when..." or "I turned off the stove"
Pedagogy and disagreements demonstrate that explaining things is difficult. The primary reason is because if it turns out the explanation is referring to a meme that the entity receiving the explanation doesn't have, then the may process fail without error. If the error is detected, it's quite possible that the meme which needs to be transmitted requires more bandwidth than what's available for receiving explanations. This assumes the missing meme can even be identified within the given constraints.
The reason "explainable AI" is difficult is because explanation is difficult. We don't know enough about how humans represent memes to cross the barrier between human and machine representations. And of course there are differences between each person's models and potentially between each machine's model that will impact the efficacy of an explanation but this can only be determined after the explanation has been evaluated. It's much easier for a machine to give an English explanation for why it perceives a shoe is red than it is for it to give an explanation for how it perceives a shoe.
I'm tired so I'll stop here. But the more useful a thing is, the more it does, the more complex it is, which makes it harder to explain, which makes it less durable.