Title: Spectral Filtering Reveals the Role of Momentum in Muon: A Two-Step Approach

Recent empirical successes of the Muon optimizer in large language model training have outpaced theoretical understanding, particularly regarding the specific function of momentum. Current scholarly attempts to explain Muon’s efficacy face a dichotomy: they either eliminate momentum to isolate the effects of spectral updates or include it without providing a rationale for its performance gains. This study resolves that disconnect by demonstrating that momentum functions as a spectral filter within the Muon framework.

By employing a structured gradient model comprising a signal and perturbation components, we establish that momentum effectively attenuates noise while maintaining the integrity of the primary signal. This process widens the spectral gap separating the signal from the perturbation. The resulting expansion of this gap serves to stabilize the singular subspaces of the matrix input to Muon’s orthogonalization phase, thereby enhancing the reliability of the final update.

Our analysis further indicates that executing momentum prior to orthogonalization yields a mathematically superior alignment with the gradient’s signal component compared to reversing this sequence or omitting momentum entirely. These theoretical findings are corroborated by experimental results across various tasks, including the pretraining of large language models. More broadly, this theoretical framework provides a foundational perspective for analyzing the utility of momentum in other matrix-based optimization algorithms.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC