Title: Preserving Plasticity in Continual Learning via Dynamical Isometry
Authors: Andries Rosseau, Robert Müller (
@deepqlearning), Ann Nowé
From the Deep Manifold view, dynamical isometry helps preserve plasticity, but it is not the source of plasticity. The deeper source comes from high-order nonlinear data forcing stacked piecewise manifolds to form ring / torus-like stationary structures.
These coupled stationary structures create elastic directions where weak perturbations can move the solution without destroying previously learned geometry.
In this sense, layer-wise isometry is an important preservation mechanism, while Deep Manifold places plasticity inside a broader geometric picture: node-cover reorientation, accumulated curvature, interconnected toroidal geometry, and eventual manifold rigidity.
#DeepManifoldInterpretation