Midnight thought -> all the big dollar is going into building infinite compute (and spoiler alert it will be there in ~5-7 years horizon).
And if we have infinite compute then we solve memory or context (2 sides of the same coin btw).
Memory dudes will argue regarding latency (i am happy getting my response 50 ms late but more accurate which comes with more context).
We can only so much optimize retrieval on an application level, real gains would be seen on a chip level and I am betting on grok, matx, nvidia etc.
So there will be many supermodels, but all of them would have same context, which will be entire world (every text written(AI generated extra), every pic you posted on insta, every ai voice recording, and gulp all the domain knowledge companies would feed adding their context and paying them for it).
And if so then what problems should we solve to get there?
The most important one that i can think of is “how will you validate context?”
Did the president really told nuclear codes or was it an AI jailbreaking it?
The concerning part is I don’t have an answer.