When Data Is Scarce: Scaling Sparse Language Models with Repeated Training
Title: Maximizing Sparse Language Models Under Data Constraints Through Repeated Training
Abstract:
While scaling laws for dense large language models (LLMs) in the context of abundant data are well-understood, the intersection of model sparsity and data scarcity remains largely unexplored. This study investigates sparse training methodologies within data-constrained environments, where the scarcity of unique tokens necessitates training over multiple epochs. Our experimental framework encompasses models with up to 1.92 billion parameters, incorporating sparsity levels as high as 93.75%. We utilized unique data budgets reaching 2.6 billion tokens and a total training volume of 41.6 billion tokens across 16 epochs. Additionally, we validated our findings by extrapolating to held-out dense-equivalent models containing up to 7.68 billion parameters.
Our key findings include:
- Scaling Laws for Data-Limited Sparse Models: We propose a new scaling law that predicts performance by modeling loss as a function of active parameters, the volume of unique tokens, data repetition rates, and sparsity levels. This model accurately forecasts outcomes across varying compute and data constraints.
- Mitigated Diminishing Returns: Sparse training extends the point at which data saturation occurs, effectively delaying the diminishing returns typically associated with repeated data exposure. Consequently, multi-epoch training proves more beneficial in these scenarios.
- Resource Optimization Trade-offs: When data volume is fixed, the sparsity level that minimizes loss is moderate, approximately 50%. However, the sparsity level that optimizes computational efficiency is higher and increases as the scale of data grows.
Collectively, these results demonstrate that sparsity serves not merely as an efficiency mechanism, but as a critical lever for optimizing scaling trade-offs when data is scarce. The code for this research is publicly available at: https://github.com/boqian333/sparse-dc-scaling.
Source: arXiv Generated at: 2026-06-02 00:00:00 UTC
