Title: Optimizing Layer-Wise Learning Rates in Linear Neural Networks: Exact Two-Step Dynamics and Scaling Laws

Abstract: This study investigates the optimal selection of learning rates for two-layer and three-layer linear neural networks tasked with learning linear target functions. We establish exact closed-form expressions for both gradients and test loss following the first and second steps of gradient descent, which allows for a precise analysis of early training dynamics. By characterizing how learning rates should be scaled under the gradient approximation during these initial steps, we demonstrate that utilizing this approximation results in a tractable surrogate loss featuring a tight and minimal approximation error. This framework facilitates the theoretical examination of layer-specific learning rates and uncovers a unique early-training regime: while unequal learning rates at the initial step can minimize test loss, equal learning rates emerge as the optimal strategy in subsequent phases. Our numerical experiments corroborate the theoretical findings, highlighting the critical role of balancing layer-wise learning rates during the early stages of training. The associated code is accessible at: https://github.com/TDCSZ327/Layer-Balancing.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC