Title: q0: Foundational Components for Hyper-Epoch Pretraining

Abstract

As computational power expands more rapidly than the availability of high-quality textual data, multi-epoch training has emerged as the prevailing standard. However, attempting to train a single model to saturation yields diminishing returns, as performance plateaus after only a few passes, leaving much of the available compute budget unused. This limitation suggests a need for a fundamental paradigm shift: moving away from optimizing a solitary model toward cultivating a diverse population of models and synthesizing their collective predictions.

To address this, we propose hyper-epoch pretraining (q0), a framework that leverages a multi-epoch budget to generate a population of varied models. The aggregated predictions from this population achieve lower validation loss than a single, highly refined model. The q0 approach relies on three essential primitives:

1. Cyclic Scheduling: By employing a cyclic schedule with anti-correlated learning rates and weight decay, the method extracts diverse models from a limited number of parallel training trajectories.
2. Chain Distillation: Each model is trained against its predecessor, ensuring that model quality compounds sequentially across the entire population.
3. Learned Prior: A prior model, trained on a held-out dataset, is used to select and weight specific members of the population for inference, allowing for flexibility across different budget constraints.

We evaluated q0 using a 1.8B-parameter model trained on 100 million tokens from FineWeb. The results demonstrate that q0 achieves performance comparable to a robust 256-epoch ensemble baseline while requiring only approximately 56 epochs—a reduction of roughly 4.6 times. When adjusted to match the ensemble size of the baseline, q0 requires only about 67 epochs (approximately 3.8 times fewer). Furthermore, q0 continues to improve beyond the baseline’s performance levels. Under the Slowrun setting, these improvements result in cumulative data efficiency gains of approximately 12.9 times, with benefits that extend to downstream benchmark tasks.

Importantly, we find that the optimal resource allocation varies depending on the total budget available. Consequently, we provide specific guidelines for distributing an epoch budget to maximize generalization, covering scenarios ranging from single-epoch training to the largest computational allocations.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC