Title: Extracting Value from Saturated Data: Utilizing Non-Binary Signals for LLM Training

Abstract:

As large language models (LLMs) continue to advance, many benchmarks and datasets previously used for their refinement have reached saturation. Prompted by this phenomenon, our study explores whether questions that already yield perfect empirical accuracy can still contribute to enhancing downstream performance. To address this, we move beyond simple binary correctness, introducing two more nuanced quality indicators: (1) pairwise self-judgments, where the model assesses the relative merit of its own generated solutions, and (2) token-level entropy, which leverages uncertainty at the token level as an indicator of solution quality. We integrated these metrics into various training algorithms and tested them on the Qwen3-1.7B-Base model. In experiments focused solely on basic arithmetic tasks, quality-driven signals boosted performance by as much as 18.6% compared to the base model, significantly surpassing standard Supervised Fine-Tuning (SFT). However, results on the GSM8K benchmark were less pronounced and highly sensitive to the specific quality signal employed. Notably, self-judgments demonstrated low concordance with a more powerful external evaluator and, in some cases, actually reduced performance below the baseline. Collectively, these findings indicate that while quality-based training can effectively mine value from saturated questions for base models, deploying these signals for more intricate tasks demands rigorous calibration and additional investigation.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC