Title: Small RL Controller, Large Language Model: RL-Guided Adaptive Sampling for Test-Time Scaling

Original: arXiv:2606.03102v1 Announce Type: new

Abstract: While test-time scaling enhances the reasoning capabilities of large language models, it significantly increases both computational expenses and latency. Current adaptive sampling techniques offer partial relief by dynamically determining when to halt the sampling process; however, these approaches generally depend on heuristic guidelines or specific distributional assumptions. To address this, we model adaptive sampling as a Markov decision process (MDP). We employ reinforcement learning (RL) to train a lightweight sampling controller capable of simultaneously optimizing for answer accuracy, latency, and computational cost. At every iteration, this controller evaluates whether to terminate sampling or to gather more samples. Our approach is resource-efficient, requiring only statistics from final answers, and can be trained and executed on CPUs. Furthermore, we demonstrate that this framework can be interpreted as the Lagrangian relaxation of a constrained optimization problem featuring explicit budget limits. Comparative experiments against robust baselines like ASC and ESC reveal that our method delivers superior trade-offs regarding answer correctness, the number of sampling rounds, and the total volume of samples needed.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC