Title: A Curvature-Adaptive $\ell_p$-Norm Framework for Deep Neural Networks: Surpassing Traditional $\ell_2$ and $\ell_\infty$ Constraints

Abstract: Conventional optimization algorithms for deep neural networks (DNNs) predominantly utilize either the $\ell_2$ or $\ell_\infty$ norms. This reliance often hinders their ability to effectively adapt to significant curvature variations across different parameter dimensions. In the initial stages of DNN training, the loss landscape typically exhibits pronounced curvature anisotropy, which gradually transitions toward flatter regions with diminished anisotropy as training progresses. Specifically, $\ell_2$-norm-based optimizers tend to be disproportionately influenced by directions with high curvature. This bias suppresses updates in lower-curvature directions, thereby impeding convergence speed. Conversely, $\ell_\infty$-norm-based optimizers frequently suffer from oscillations in flatter areas, a consequence of applying uniform magnitude updates across coordinates. To mitigate the limitations inherent in these two extremes, we introduce a novel $\ell_p$-norm approach featuring a dynamic parameter $p$. By integrating this scheme into Stochastic Gradient Descent (SGD) and Stochastic Gradient Descent with Momentum (SGDM), we derive two new optimizers, denoted as $\ell_p$-SGD (LPSGD) and $\ell_p$-SGDM (LPSGDM), which demonstrate superior generalization capabilities. In the early training phases, a large $p$ value ($p>2$) is employed to counteract the overwhelming influence of high-curvature directions. Subsequently, $p$ is gradually reduced toward 2, facilitating more stable and precise updates; this decay mechanism is inspired by the cosine annealing strategy. We provide theoretical validation for these algorithms, proving that both LPSGD and LPSGDM attain an $O(T^{-1/2})$ convergence rate in nonconvex settings. Our empirical evaluation covers standard benchmark datasets—CIFAR-10, CIFAR-100, and ImageNet-1K—utilizing various DNN architectures, including VGG-11, ResNet-18, and ResNet-50.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC