Title: Thinking Economically: A Hierarchical Framework for Adaptive-Complexity Reasoning in LLMs

Original: arXiv:2606.01168v1 Announce Type: new Abstract: Chain-of-Thought (CoT) has significantly enhanced LLM reasoning, yet often incurs substantial computational overhead due to "overthinking": generating excessively long rationales without commensurate accuracy gains. Existing efficiency methods typically apply uniform compression, which overlooks a critical observation that reasoning complexity is heterogeneous at two distinct granularity: across different problems and within individual reasoning steps. This motivates our principle of Thinking Economically: intelligently allocating computational resources based on intrinsic task and step demands rather than pursuing uniform brevity. We propose Hierarchical Adaptive Budgeter (HAB), a training framework that operationalizes this principle through coarse-to-fine budgeting. At the inter-step level, HAB predicts the optimal reasoning depth for each problem. At the intra-step level, HAB learns step-specific token budgeting signals from PPL-derived step comparisons and an adaptive Pareto optimization objective that captures the local quality-efficiency trade-off, while a Fisher Information-based pruner further provides fine-grained training-time guidance, thereby encouraging the generator to internalize more economical reasoning patterns. Experiments on GSM8K and MATH500 show that HAB not only surpasses standard CoT in accuracy but also reduces token usage, achieving a stronger performance-efficiency trade-off than the compared baselines.

Rewrite: arXiv:2606.01168v1 Announce Type: new Abstract: While Chain-of-Thought (CoT) techniques have markedly improved the reasoning capabilities of Large Language Models (LLMs), they frequently suffer from high computational costs stemming from "overthinking"—a phenomenon where models produce unnecessarily verbose explanations that do not yield proportional improvements in accuracy. Current approaches to efficiency often rely on uniform compression strategies, ignoring the fact that reasoning complexity varies significantly at two levels of granularity: across distinct problems and among individual steps within a single reasoning process. This discrepancy underscores the need for a "Thinking Economically" approach, which advocates for the intelligent distribution of computational resources according to the specific demands of the task and reasoning steps, rather than enforcing a one-size-fits-all brevity. To implement this concept, we introduce the Hierarchical Adaptive Budgeter (HAB), a training framework that employs a coarse-to-fine budgeting strategy. At the inter-step level, HAB determines the ideal reasoning depth for each problem. At the intra-step level, it derives step-specific token budgeting cues through comparisons based on perplexity (PPL) and an adaptive Pareto optimization goal that balances local quality and efficiency. Additionally, a pruner utilizing Fisher Information offers detailed guidance during training, prompting the generator to adopt more resource-efficient reasoning habits. Evaluations on the GSM8K and MATH500 datasets demonstrate that HAB outperforms standard CoT in both accuracy and token reduction, delivering a superior balance between performance and efficiency compared to existing baseline methods.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC