Amongst my favourite research directions this year: understanding model complexity and its link to generalization and intelligence. Progress here could mean leaner models, versatile representations, and less reliance on data/energy. Excited that we’re off to the races on this!
I’m pleased to announce our work which studies complexity phase transitions in neural networks! We track the Kolmogorov complexity of networks as they “grok”, and find a characteristic rise and fall of complexity, corresponding to memorization followed by generalization.
🧵