Great post. I think there is much more to unpack behind the spectrum-preserving update rule in Pion (spherelab.ai/pion). Jianlin derives a different spectrum-preserving update rule from a steepest-descent perspective, leading to an alterantive orthogonalization (msign).

In PEFT-Arena (spherelab.ai/PEFT-Arena), we found a “free lunch” that improves both adaptation performance and preservation of general capabilities across PEFT methods, including full-parameter finetuning. The trick is simple: interpolate the weights and choose a midpoint between the fine-tuned model and the pretrained model. This can be useful in practice. For Orthogonal Finetuning (OFT), the best interpolation is not linear. It should respect the orthogonal geometry, so the interpolation is performed within the orthogonal group (the computation is still very simple).

Quite inspiring. Optimizer design seems increasingly architecture-driven, from vector-based to matrix-based, and ultimately architecture-aware.

I find this project particularly elegant because it addresses a simple yet practically important question: should momentum be applied before or after the orthogonalization step? We study this question through the lens of spectral filtering and show that applying momentum before orthogonalization acts as a denoiser and can be provably better than applying momentum afterward.