Title: A Task-Centric Theory for Iterative Self-Improvement with Easy-to-Hard Curricula

Abstract: Iterative self-improvement involves fine-tuning an autoregressive large language model (LLM) using reward-verified outputs that the model generates itself. Despite the empirical success of this approach, the theoretical underpinnings of such a generative, iterative process in practical, finite-sample contexts remain underexplored. To address this gap, we model each iteration of self-improvement as maximum-likelihood fine-tuning on a distribution filtered by rewards, establishing finite-sample guarantees for expected reward performance. Our analysis uncovers an explicit feedback mechanism wherein superior models are able to accept more data in each iteration, thereby enabling continuous self-improvement and accounting for the eventual saturation of gains. By adopting a task-centric perspective that incorporates reasoning tasks of varying difficulty levels, we demonstrate that easy-to-hard curricula offer provably superior guarantees compared to training on static task mixtures, provided specific conditions regarding model initialization, task complexity, and sample budget are met. These theoretical insights are corroborated by Monte-Carlo simulations and experiments involving a synthetic graph-based reasoning task alongside several standard mathematical reasoning benchmarks.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC