PreconditioningSpectral norm constraintsmuP transferLarge-batch training dynamics
Second-Order Signals in Large-Scale Optimization
1 min readLearnJul 17, 2026
Key Takeaway for Students
Optimizers like Shampoo, SOAP and Muon exploit curvature or spectral structure to beat AdamW at large batch sizes, at the cost of memory and communication. Their interaction with muP-style scaling rules is an open question.
Reading in Plain English mode — toggle below to switch
Optimizers like Shampoo, SOAP and Muon exploit curvature or spectral structure to beat AdamW at large batch sizes, at the cost of memory and communication. Their interaction with muP-style scaling rules is an open question.