Title: BLISS: A Lightweight Bilevel Influence Scoring Method for Data Selection in Language Model Pretraining

Abstract:

Optimizing data selection is crucial for the pretraining of large language models (LLMs), as it boosts efficiency and enhances generalization capabilities in downstream applications. Current methods, however, typically depend on external pretrained models, which complicates the isolation of data selection effects from those of the oracle models. Furthermore, due to the immense computational costs associated with full-scale LLM pretraining, most existing techniques neglect the long-term consequences of data selection when models are trained to convergence.

To address these challenges, we present BLISS (BileveL Influence Scoring method for data Selection), a novel, lightweight data selection framework. Unlike prior approaches, BLISS operates entirely from scratch, requiring no external pretrained oracle models, and explicitly incorporates the long-term impact of the chosen data. The method utilizes a small proxy model as a substitute for the LLM, alongside a score model designed to estimate the long-term influence of training samples assuming the proxy model reaches convergence.

We frame data selection as a bilevel optimization problem. In this setup, the upper-level objective trains the score model to assign importance weights to training samples, while the lower-level objective minimizes the weighted training loss of the proxy model until convergence. The goal is to ensure that this process yields the best possible validation performance. Once the score model is optimized, it generates influence scores for the entire dataset, facilitating the efficient identification of high-quality samples for LLM pretraining.

We evaluated BLISS by pretraining Pythia models (410M, 1B, and 2.8B parameters) and an LLaMA-0.5B model using selected subsets of the C4 dataset. Results indicate that BLISS outperforms existing methods across various downstream tasks. Specifically, in the 1B model configuration, BLISS achieves a 1.7x speedup in reaching performance levels comparable to state-of-the-art techniques.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC