Refreshing deep learning with a 400 textbook: "The Principles of Deep Learning Theory" Covers: - An "effective theory" framework for deep neural networks - Criticality, chaos, and signal propagation at initialization - Neural Tangent Kernels (NTKs) and kernel learning - Bayesian learning and finite-width effects - Why deep networks learn features instead of behaving like kernels

Wow, this is interesting.. @Stanford researchers put a common assumption to the test: large models need only “high-quality” filtered training data. What if the best filter is no filter at all? They compared full Common Crawl data with heavily filtered versions of it and got surprising results: 1. Filtering can help with small compute budgets, because models can't learn from everything well. 2. However, as models get larger and train longer, the full, unfiltered dataset becomes the winner. Large models handle messy data better than expected – low-quality text, irrelevant text, or some “junk” data are not a big deal; these models can tolerate them. And they can even extract useful signal from data that looks poor. These facts transform general rules: → Filtering helps when compute is limited. But when compute is very large, removing too much data may throw away useful information. This also connects with the concept of "bitter lesson": at large scale, simple scaling often beats clever human design. But the final choice depends on your constraints and preferences – would you rather increase compute costs or put more resources and time into filtering? Interesting to see your answers 👀