Title: Pseudospectral Limits on Transient Growth in Coupled Gradient Descent

Abstract:

Coupled gradient descent, a framework where the update of one parameter block relies on another, serves as the foundation for bilevel optimization, two-time-scale stochastic approximation, and adversarial training. Although the asymptotic stability of systems with block-triangular coupled Jacobians is determined by the spectral radii of their diagonal components, transient amplification prior to convergence can become arbitrarily large owing to non-normality. This study introduces a precise pseudospectral theory for these block-triangular Jacobians. We demonstrate that if the diagonal blocks are symmetric and possess spectral radii no greater than $\gamma < 1$, the Kreiss constant is bounded by $K(J) \leq 2/(1-\gamma) + |C|/(4(1-\gamma))$, and we further derive corresponding minimax lower bounds to establish tightness. The analysis identifies the critical coupling threshold leading to spectral instability and generalizes to nearly self-referential systems through a Neumann-series perturbation approach. Consequently, we derive a finite-horizon iteration-complexity bound of $O(K(J)^2 \log(1/\delta))$ for stochastic coupled descent. Interpreted as scaling laws for non-stationary two-time-scale optimization, these findings reveal a non-asymptotic, instance-dependent phase in high-dimensional learning dynamics that remains undetected by standard spectral-radius analysis. Empirical validation is provided through experiments involving linear-quadratic problems, comparisons based on Integral Quadratic Constraints (IQC), and neural-network training.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC