Title: Investigating the Stability of Sharpness-Aware Minimization

Abstract:

Sharpness-Aware Minimization (SAM) is a widely adopted training technique in deep learning that aims to locate flat minima, thereby delivering state-of-the-art results across numerous fields. Unlike standard approaches that optimize the loss associated with current parameters, SAM focuses on minimizing the worst-case loss within the immediate vicinity of the parameter space. This study examines the convergence instability exhibited by SAM when approaching saddle points. By applying the qualitative theory of dynamical systems, we elucidate the mechanisms by which SAM becomes trapped at these points and provide a theoretical proof that saddle points can function as attractors under SAM dynamics. Furthermore, we extend this analysis to stochastic dynamical systems by deriving the diffusion characteristics of SAM. Our findings indicate that SAM’s diffusion performance regarding saddle point escape is inferior to that of vanilla gradient descent. Lastly, we highlight that commonly neglected training strategies, specifically momentum and batch size, play a crucial role in alleviating convergence instability and enhancing generalization. The validity of our theoretical insights and empirical observations is confirmed through extensive experiments on established optimization problems and benchmark tasks.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC