Title: KromHC: Leveraging Kronecker-Product Residual Matrices for Manifold-Constrained Hyper-Connections

Abstract: While Hyper-Connections (HC) have demonstrated significant success in neural networks (NN), they have also exposed problems concerning scalability limitations and training instability. The Manifold-Constrained Hyper-Connections (mHC) framework attempts to resolve these issues by projecting the residual connection space onto a Birkhoff polytope. However, this approach encounters two primary difficulties: first, its iterative Sinkhorn-Knopp (SK) algorithm fails to consistently produce strictly doubly stochastic residual matrices; second, mHC suffers from an excessive parameter complexity of $O(n^3C)$, where $n$ represents the residual stream width and $C$ denotes the feature dimension. Although the recently introduced mHC-lite utilizes the Birkhoff-von-Neumann theorem to reparametrize the residual matrix and ensure double stochasticity, it introduces a factorial explosion in parameter complexity, reaching $O \left( nC \cdot n! \right)$. To overcome these obstacles, we introduce KromHC, a method that parametrizes the residual matrix in mHC using Kronecker products of smaller doubly stochastic matrices. By applying manifold constraints across the factor residual matrices along each mode of the tensorized residual stream, KromHC ensures the exact double stochasticity of the residual matrices and lowers the parameter complexity to just $O(n^2C) Empirical results indicate that KromHC performs on par with or better than other state-of-the-art (SOTA) mHC variants, all while utilizing substantially fewer trainable parameters. The code is available at https://github.com/wz1119/KromHC.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC