I completely agree. There is also the false differentiation between struggle and enjoyment.
First of all, reading something written by somebody else, we must attempt to suppress our expectations, and open ourselves to what is before us, silencing ourselves, forgetting ourselves. Because reading, and the appreciation of art in general, is an act of transcendence.
Secondly, it's as simple as this: some (by no means all) great books can be difficult to read, or difficult at least to endure, two reasons for the inability to overcome which include: 1) not silencing ourselves, and 2) not knowing or not trying to understand the author's purpose.
Some books gave me a lot of work, and at times I didn't know what to make of them, or they contradicted ideas I'd formed too quickly. Ulysses, Eliot's work, Moby-Dick, Thomas Mann's novels, are a few examples. Yet they are also among my all-time favourite works, and there have been few such rewarding experiences in my life.
And even those works or authors whose prose is incredibly easy to follow, or at least not too complicated - such as Tolstoy, Dostoyevsky, Tolkien, etc. - involve depth that goes beyond all sorts of superficial analyses and comments that I've seen on this platform.
The idea, also, that everybody can just interpret a book in whatever way they wish, something true only in a restricted way, is much abused and misunderstood. For instance, if you read Dostoyevsky's Demons and what you take from it is that it's cool to be a nihilist, then you are objectively wrong, and completely unaware of the author's purpose and philosophy. It doesn't mean you have to be an Orthodox Christian to read it, but you also can't make claims regarding the book that are absolutely contrary to its intention. I love the Upanishads, but I would never call them Christian. Everybody will agree that it would be absurd to do so.
Much more could be said, and I may write an article about this soon - but what should be clear is that not everything in a book will be amusing, and that we should approach every work of art in a spirit of openness - even one produced by someone whose convictions and experiences are similar to ours. For we are all, though profoundly similar, radically different and unique - and appreciating art must be also an act of humility. Do you think it right that in a conversation you should make everything about yourself? Then still less when you are a reader, a listener, or a viewer.
A curious modern superstition: that books exist primarily to amuse us. There are certainly books one should abandon. Life is short. But there are others which must be endured before they can be loved, rather as one acquires a taste for olives, or Mahler.