Title: Theoretical Analysis Shows Loop-Structured Transformers with Layer Normalization Naturally Implement the Power Method

Recent studies indicate that the exceptional performance of Transformers across various domains stems partly from their capacity to acquire and execute algorithmic procedures. Despite this, the mechanisms by which these models internalize such algorithms remain poorly understood, particularly when Layer Normalization (LN) is involved. This study addresses this gap by utilizing principal component prediction as a specific framework to analyze the training dynamics of LN-equipped Transformers.

We demonstrate that a linear Transformer featuring a looped architecture and LN, when optimized via gradient descent, converges to a solution that effectively executes the power method. In this configuration, each self-attention layer corresponds to a single power iteration. Crucially, this behavior emerges without explicit supervision to implement the algorithm; the model is trained solely for the task of principal component prediction. This outcome highlights an "algorithmic implicit bias" inherent to looped Transformers with LN. While principal-component prediction can theoretically be accomplished through numerous distinct mechanisms, gradient descent specifically favors the solution that mirrors the power method.

Furthermore, we establish a clear distinction between Transformer architectures with and without LN. Our analysis shows that even when provided with layerwise guidance derived from power iterations, a Transformer lacking LN fails to learn the power method exactly. In contrast, the LN-enabled counterpart succeeds, resulting in a provable performance advantage in principal component prediction tasks. To our knowledge, this work offers the first theoretical examination of the training dynamics for both single-layer and looped Transformers incorporating LN, offering new insights into the functional role of normalization in these models.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC