Title: ReasonBENCH: Benchmarking the (In)Stability of LLM Reasoning

Abstract:

While benchmark scores for Large Language Model (LLM) reasoning systems are typically presented as singular metrics, the reality is far more variable. Even when employing identical models, strategies, and tasks under deterministic conditions (greedy decoding with T = 0), repeated executions can yield significantly different outcomes and resource costs. This variability is not merely a statistical artifact; it poses a substantial risk to accurate system ranking. For instance, the top-performing strategy secures victory in only 77% of direct comparisons against its closest rival, implying that a single observed score can inadvertently misrepresent system capabilities.

To address this, we present ReasonBench, a comprehensive benchmarking suite that captures 30 independent trials across 10 distinct reasoning strategies, 12 models, and 6 tasks. By treating both quality and cost as distributions rather than static point estimates, our analysis reveals that this variance is structured, not random. We propose a two-component taxonomy to explain these fluctuations: "Global Noise," which accounts for uneven performance across different benchmarks, and "Run Noise," which captures stochasticity within a single benchmark. This framework demonstrates that while strategy architecture dictates stability profiles, models and strategies independently influence orthogonal aspects of the performance distribution.

Furthermore, a hierarchical decomposition analysis indicates that 75% of score variance is attributable to the structure of the benchmark, the system, and specific items. The remaining residual variance is often overlooked by single-run evaluations. Additionally, we observe an asymmetric decoupling of cost and quality: inexpensive methods are structurally protected against simultaneous failures in cost and quality, whereas costly methods remain vulnerable to such failures regardless of their accuracy. These insights confirm that instability is an intrinsic characteristic of reasoning systems, underscoring the necessity of adopting distribution-aware evaluation as the standard practice.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC