Title: An Asymptotic Theory of Chain-of-Thought in In-Context Learning

Abstract:

While Chain-of-Thought (CoT) prompting is a prevalent strategy for inducing multi-step reasoning in large language models by producing intermediate logical steps during inference, the scaling laws governing generalization relative to CoT depth are not well characterized. To investigate this gap, we examine a theoretically tractable framework for CoT-based in-context weight prediction within linear regression. In this model, reasoning conducted at test time is formalized as an iterative process that refines the estimation of weight parameters. By employing random matrix theory under high-dimensional asymptotic conditions, we derive a precise mathematical expression for generalization error, explicitly linking it to reasoning depth, the volume of pretraining data, and context length. Our theoretical findings identify a distinct phase transition that delineates regimes of exponential and polynomial performance gains, as well as points of saturation and "overthinking." Additionally, we define the scaling relationship for optimal reasoning depth. The study demonstrates that increased reasoning depth yields the greatest benefits when supported by ample pretraining and rich in-context data; conversely, sparse pretraining or limited context increases the risk of error amplification or performance saturation during extended reasoning. These theoretical predictions are corroborated through experiments involving fully learned linear attention and softmax attention architectures. Ultimately, this work offers a cohesive theoretical explanation for the impact of test-time CoT depth on model generalization.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC