Title: What Do Students Learn? A Feature-Level Analysis of Dark Knowledge

Original: arXiv:2606.03052v1 Announce Type: new

Abstract: Although Knowledge Distillation (KD) is a widely recognized technique for compressing models, the specific processes through which student networks acquire feature representations have not been thoroughly investigated. This study employs the Interaction Tensor framework to examine feature learning within student models. Our findings indicate that successful KD functions as a regularization mechanism, filtering out low-frequency, sample-specific attributes and prompting the student to depend on a concise collection of highly versatile features. Importantly, we identify that the dataset-level confusion matrix holds structural insights similar to the teacher’s "Dark Knowledge." Building on this discovery, we introduce Confusion Distillation (CD), a novel, teacher-free self-distillation approach. CD leverages the model’s own changing confusion patterns as dynamic soft targets. In experiments on CIFAR-100 using ResNet-34 and ResNet-50 architectures, CD delivers competitive results, surpassing current self-distillation techniques such as CS-KD and PS-KD by 1.2%, while providing a more computationally efficient solution compared to traditional KD methods.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC