Title: Forecasting Inference-Time Scaling Benefits Using Labeled Validation-Set Output Metrics

Original: arXiv:2606.02981v1 Announce Type: new Abstract: Best-of-$N$ inference scaling (drawing $N$ candidate answers from a language model and returning the one a reward model ranks highest) improves accuracy by an amount that varies across models, but predicting that amount in advance currently requires running the procedure end-to-end. Prior work links cheap statistics of a model's sampled outputs and validation-set correctness (how often samples agree, how diverse they are, how confident the model is, and where correct samples appear) to model behavior, but does not isolate which of these form a stable, compact predictor of best-of-$N$ gain. We fit ridge predictors on features computed from a single labeled validation-set sampling pass, use bootstrap-Lasso as a stability analysis of the candidate feature set, and give a concentration analysis with an explicit linear-approximation residual. Across three base-model families, six post-training methods, and math and reasoning task domains, the stability analysis identifies a strict three-feature core spanning prompt-level agreement spread, label-assisted first-correct-sample position, and completion-length variance; a compact ridge predictor built from this core plus an entropy add-on reaches Spearman $\rho = 0.90$ with actual best-of-$N$ gain under a reward-model verifier. The intended use is labeled validation-set screening of candidate configurations before paying the full reward-model scoring cost.

Rewritten: arXiv:2606.02981v1 Announce Type: new Abstract: The accuracy improvements achieved through best-of-$N$ inference scaling—where a language model generates $N$ candidate responses and a reward model selects the highest-ranked one—exhibit significant variability across different architectures. Currently, forecasting this performance boost necessitates executing the entire procedure. While previous research has correlated inexpensive metrics derived from sampled outputs and validation-set accuracy, such as sample consensus, diversity, confidence levels, and the placement of correct answers, with model behavior, it has failed to pinpoint which specific metrics constitute a robust and concise predictor of best-of-$N$ gains. To address this, we trained ridge regression models using features extracted from a single pass of labeled validation-set sampling. We employed bootstrap-Lasso for stability analysis of the candidate feature pool and provided a concentration analysis featuring an explicit linear-approximation residual. Our investigation, covering three base-model families, six post-training techniques, and both mathematical and reasoning task domains, revealed a core set of three stable features: the spread of prompt-level agreement, the position of the first correct sample assisted by labels, and the variance in completion length. By combining this core with an entropy component, we constructed a compact ridge predictor that achieved a Spearman $\rho$ of 0.90 against the actual best-of-$N$ gains verified by a reward model. This approach aims to facilitate the screening of candidate configurations using labeled validation sets, thereby avoiding the high costs associated with full reward-model scoring.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC