Title: The Paradox of Outcome Optimization: A Causal Information-Theoretic Bound on Reasoning Shortcuts in LLMs

Abstract:

Large Language Models (LLMs) trained using outcome-based Reinforcement Learning (RL) often encounter a significant vulnerability: while they excel on in-distribution benchmarks, their reasoning skills prove fragile when applied to out-of-distribution (OOD) tasks. We label this specific failure mode "Reward-Induced Manifold Collapse." To elucidate this paradox, we construct a theoretical framework that integrates Structural Causal Models (SCM) with the Information Bottleneck (IB) principle.

In our analysis, we characterize reasoning as a high-complexity causal mechanism, whereas shortcut learning is defined as the utilization of low-complexity spurious correlations. We demonstrate that the implicit inductive bias inherent in Stochastic Gradient Descent (SGD) drives outcome-optimized models toward shortcut solutions whenever the training distribution permits the "Markovian Screening" of the underlying causal structure.

Furthermore, we derive a novel generalization bound predicated on the Semantic Coverage Measure ($\eta$) instead of traditional sample size metrics. This derivation explains why merely scaling data within homogeneous distributions is insufficient to rectify reasoning deficiencies. Additionally, we illustrate how Process Reward Models (PRMs) act as Topological Filters; by imposing step-wise mutual information constraints, these filters render the low-complexity shortcut manifold inadmissible. Collectively, these findings offer a rigorous mathematical justification for the necessity of process supervision in LLMs, extending beyond the scope of simple credit assignment.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC