Title: Challenges in Learning a Meta-Network for Training Data Selection

Abstract:

While synthetic data is becoming a staple for training neural networks, its utility is often constrained by distributional discrepancies with real-world data, particularly when applied without careful curation. To address this, a prevalent approach involves determining data weights through bi-level optimization, a process we term Meta-learning for Training-data Selection (MTS). However, empirical evidence suggests that MTS frequently underperforms relative to theoretical expectations.

In this work, we pinpoint two primary hurdles in the effective training of MTS: an insufficient gradient signal-to-noise ratio (GSNR), which hinders optimization, and an absence of features that meaningfully correlate with data quality. Through mathematical analysis, we examine the behavior of normalized data weights and elucidate the connection between varying data quality and low GSNR. Our findings indicate that a straightforward yet potent remedy is to increase the batch size. Additionally, we introduce a collection of informative features designed to reflect both the position of training samples within their distributions and their training dynamics. Validated across four distinct benchmarks, our approach yields consistent performance enhancements, delivering average improvements of 5.49% compared to training without selection and 2.89% over the most robust baseline.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC