Title: A Rigorous Approach to Rescaling ReLU Neural Networks via Path-Conditioned Training

Abstract:

Although significant algorithmic progress has been made, there remains a lack of theoretically sound methods for utilizing the established rescaling symmetries inherent in ReLU neural network parameters. Although different weight configurations that are appropriately rescaled yield identical functions, their training dynamics can vary significantly. To provide a novel perspective on harnessing this behavior, we extend the recent path-lifting framework, which offers a concise factorization of ReLU networks. We propose a geometrically inspired criterion for rescaling network parameters; minimizing this criterion results in a conditioning strategy that aligns the kernel within the path-lifting space to a predefined reference. We also develop an efficient algorithm to execute this alignment. Furthermore, we examine how the architecture and initialization scale collectively influence the output of our method under random network initialization. Our numerical experiments demonstrate the method's potential to accelerate the training process.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC