Title: Geometric Properties of Stationary Plateaus in Two-Layer Neural Networks

Abstract: This study examines the geometric architecture of stationary plateaus within the loss landscapes of two-layer neural networks utilizing smooth activation functions. Central to our investigation is the "neuron splitting" phenomenon, a process wherein replicating a hidden neuron generates an affine manifold of stationary points within an expanded network topology. We present a thorough taxonomy of every stationary point residing on these plateaus, establishing the specific criteria that classify them as either local minima or saddle points. This characterization relies on a per-neuron curvature metric we define as the "inner Hessian." Our analysis demonstrates that the local geometry of the plateau is governed by the interplay between the definiteness of this inner Hessian and the selection of splitting coefficients. Specifically, we demonstrate that splitting a local minimum can result in a plateau composed of mixed local minima and saddle points, or entirely of saddle points, with a distinct sure-saddle region identified under mild assumptions. Conversely, splitting a saddle point invariably yields a plateau consisting solely of saddle points. These findings consolidate and broaden previous landscape analyses, clarifying the mechanisms by which model expansion influences the nature of stationary points. Ultimately, these results provide novel geometric perspectives on the impact of width expansion and reparameterization in neural networks.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC