Title: Leveraging Weight-Space Symmetries to Approximate Curvature

Abstract:

Estimating the curvature of a loss function is a critical component of numerous machine learning algorithms; however, performing this task at the scale of contemporary deep neural networks remains a significant challenge. Notably, prior research has overlooked the curvature constraints inherent in the well-established weight-space symmetries found within loss landscapes. To address this gap, we employ analytical averaging over group actions that preserve loss invariance. This approach allows us to derive structured Hessian approximations using only single gradients, which are computationally efficient to estimate, store, and invert.

The user-defined symmetry group serves as the primary mechanism for balancing the trade-off between computational expense and approximation precision. Furthermore, this framework offers a cohesive theoretical perspective on existing techniques; for instance, selecting a particular symmetry group yields curvature estimates akin to those produced by Shampoo and Muon. We demonstrate the efficacy of our approach across various network architectures and apply it to second-order optimization benchmarks, including the training of a small language model. Beyond optimization, our curvature estimation framework holds potential for broader machine learning applications, such as uncertainty quantification, continual learning, model compression and pruning, and training data attribution.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC