A critical initialization for biological neural networks Spontaneous brain activity is often treated as noise: the background hum of a nervous system waiting for a task. But large-scale recordings in mice have shown something more structured. Even in darkness, without explicit stimuli, thousands of neurons display coordinated activity patterns that extend across the brain and persist far longer than the fast biophysical timescales of individual neurons. Marius Pachitariu and coauthors ask a simple question: could this macroscopic structure emerge from a simple kind of network initialization? Their answer connects neuroscience, random matrix theory and machine learning. They model spontaneous neural activity as linear dynamics governed by a random connectivity matrix, stabilized by a global inhibitory-like normalization. When this matrix is symmetric and critically normalized, with its largest eigenvalue very close to one, the network naturally produces high-dimensional activity modes with a power-law covariance spectrum. This is not just a mathematical curiosity. The same spectral structure appears in large-scale mouse recordings from cortex and brainwide Neuropixels data, with power-law exponents around 0.7–0.85. Hippocampal CA1 is the striking exception: its activity looks less correlated, closer to an efficient, high-capacity code for information storage. The ML perspective is especially interesting. In artificial neural networks, initialization is often treated as a technical detail: Xavier, He, orthogonal schemes, and so on. But this paper reframes initialization as a computational substrate. A critically initialized recurrent system can generate slow, global, high-dimensional modes before task-specific learning. In simulations, these dynamics support time-dependent computations, including zero-shot working memory tasks. The biological implication is powerful: spontaneous activity may not be random noise, but a preconfigured dynamical scaffold on which learning and computation can operate. The brain may start from an initialization already close to useful temporal memory, with learning then shaping readouts or task-specific pathways. For R&D teams building ML systems in drug discovery, materials development, energy research or biotechnology, the lesson is broader than neuroscience. Initialization, architecture and dynamics define what kinds of scientific signals a model can preserve, combine and retrieve before training. In applied research pipelines where data are scarce, noisy and time-dependent, designing the right dynamical substrate may be as important as choosing the loss function. Source: Pachitariu et al., Nature (2026) — CC BY 4.0 | doi.org/10.1038/s41586-026-1…

Running something called "The Rosetta Stone Experiment" training 4 transformers from scratch right now. same architecture, same hyperparameters, completely different data which is english text, python code, protein sequences, and music. the question is whether the same internal circuits emerge across all four. things like induction heads, the OV and QK weight structure, spectral patterns in the weight matrices. the hypothesis: gradient descent keeps rediscovering the same computational primitives regardless of what it's trained on. if true, these aren't language-specific tricks they're something more fundamental about how transformers process sequences and if its false then oh well :p still fun isnt it? #machinelearning #mlresearch #airesearch #iamsotired