"100 million words context window is already possible, which is roughly what a human hears in a lifetime. Inference support is the only bottleneck to achieve it.
And AI Models actually do learn during the context window, without changing the weights."
~ Anthropic CEO Dario Amodei
(On the 2nd point, there was this brilliant Google Paper published last week that says, LLMs can learn in context from examples in the prompt, can pick up new patterns while answering, yet their stored weights never change.)
---
From 'Alex Kantrowitz' YT Channel (Full Video link in comment)