Title: Moving Past Structural Symmetries: Linear Mode Connectivity Through the Lens of Neuron Identifiability

Abstract:

In the realm of deep learning, prominent observations such as structured training dynamics and linear mode connectivity are frequently linked to parameter symmetries—transformations that preserve the function actually realized by the model. However, despite the increasing focus on these symmetries, the precise relationship among parameters, data, and internal representations has not been thoroughly examined. To address this gap, we introduce a theoretical framework centered on "effective function classes," defined as the collection of functions a neuron can generate across its input support, along with the associated norm costs for achieving them. We subsequently define effective symmetry breaking through the concept of neuron identifiability, observed across distinct training runs. Our investigation reveals that even in models lacking structural asymmetry, neural networks can support extensive families of nearly equivalent solutions. Furthermore, we demonstrate that neuron identifiability facilitates the merging of representations without the need for prior alignment, and we delineate the conditions under which this merging process yields a linear, low-loss trajectory. These results underscore the significant influence that effective function classes exert on the structure of the loss landscape.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC