Title: Adaptive Sharpness-Aware Minimization with a Polyak-type Step size: A Theory-Grounded Scheduler

Abstract:

Sharpness-Aware Minimization (SAM) has emerged as a highly effective and widely utilized optimizer for machine learning model training. By directly targeting the minimization of loss landscape sharpness, SAM frequently enhances generalization capabilities while maintaining robust empirical results. Nevertheless, like many training algorithms, SAM and its derivatives remain highly sensitive to learning rate selection, a parameter usually determined via rigorous hyperparameter search or fixed scheduling policies.

Driven by recent findings regarding the efficacy of stochastic Polyak step sizes in Stochastic Gradient Descent (SGD), this study introduces Polyak schedulers specifically adapted for SAM-style updates. These adaptations result in new adaptive algorithms applicable to both deterministic and stochastic environments. Our theoretical analysis demonstrates that, in smooth scenarios, the proposed method achieves linear convergence for strongly convex objectives and an $\mathcal{O}(1/T)$ convergence rate for convex objectives within the deterministic framework. In stochastic contexts, we provide equivalent convergence guarantees, albeit up to a vicinity of the optimal solution.

Empirical evaluations indicate that these novel Polyak schedulers deliver performance on par with or superior to meticulously tuned SAM baselines. Crucially, this approach significantly alleviates the burden of learning-rate tuning.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC