🎟️We all know The Lottery Ticket Hypothesis raveled the fact that there exist sub/sparse networks within the given dense networks that if trained can be as good if not better than its dense counterparts.
But the fact that field like Reservoir Compute exist and works unreasonably well makes me wonder how much of the performance in those sub/sparse networks is from the updated weights and how much of it is directly from the structure?
Papers like: "What’s Hidden in a Randomly Weighted Neural Network?" & "Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask" demonstrated that connection masks alone can contribute significantly to performance, with signs and zeros in weights playing complementary roles.
This leads to a significant finding that we need only fine-tune a tiny fraction of all parameters to achieve performance as good as full fine-tuning, this finding not only helps model saves compute, but also shows us we can do so while mitigate the catastrophic forgetting issue, two birds one stone indeed, as we can see in papers like:
"Sparse is Enough in Fine-tuning Pre-trained Large Language Models"
"LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptation"
"Continual Learning via Sparse Memory Finetuning"
On the surface, seems training/fine-tuning sparse model is the best we can do to achieve compute efficiency & some level of success in Continual Learning, what else can we do to further improve it?
Well, what about we just use the large & completely random networks directly and only update a tiny part of the weights where it's most useful?
Most researcher would think there is no way in hell this can work, but it just might! Since the paper "The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training" has shown when a model is large enough, all tickets/subnets become winning tickets!
I'll try to find relevant papers for it first, if anyone knows anything like then please share it to me, if not then I guess I'll find time to conduct this research myself, for Truth🚀