Title: Reintegrating Hard Labels: A Strategy to Counteract Local Semantic Drift

Abstract:

In the realms of knowledge transfer and large-scale dataset distillation, utilizing soft labels from teacher models has become standard practice, as seen in methods like SRe2L and LPLD. However, these approaches face significant challenges when the number of image crops is restricted to mitigate the high costs associated with storing precomputed soft labels. This limitation often triggers severe local semantic drift, where visually ambiguous crops cause soft supervision to diverge from the image-level ground truth. Such divergence results in entrenched errors and a mismatch between training and testing distributions.

This study revisits the underappreciated function of hard labels, demonstrating that their proper integration serves as a content-invariant semantic anchor capable of calibrating this drift. We provide a theoretical analysis of how drift arises under sparse soft-label supervision and prove that combining hard and soft labels realigns visual content with semantic supervision. Leveraging this finding, we introduce a novel training framework called Hard Label for Alleviating Local Semantic Drift (HALD). This paradigm employs hard labels as intermediate corrective signals while retaining the detailed advantages of soft labels.

Comprehensive experiments conducted on large-scale classification benchmarks and dataset distillation tasks reveal consistent enhancements in generalization. Notably, on ImageNet-1K, our approach achieves an accuracy of 42.7% while requiring only 285M of soft-label storage—a 100-fold reduction compared to previous methods. This performance surpasses the prior state-of-the-art method, LPLD, by 9.0%.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC