Title: ASKD-Whisper: Adaptive Self-knowledge Distillation for Efficient and Low-Latency Automatic Speech Recognition

Abstract:

Knowledge distillation (KD) stands out as a premier methodology for transforming expansive foundation models into practical, deployable structures. Within the field of Automatic Speech Recognition (ASR), earlier research has largely concentrated on compelling student models to rigidly replicate the predictive distributions of their much larger teacher counterparts. Nevertheless, this fixed reliance frequently creates a fundamental compromise: although students quickly absorb foundational linguistic patterns, they also absorb the teacher’s domain-specific limitations and excessive confidence in hallucinations. Consequently, this leads to a significant deterioration in the model’s ability to generalize outside of its training distribution.

To address these challenges, we introduce Adaptive Self-Knowledge Distillation (ASKD), a dynamic curriculum-based approach. ASKD works by progressively reducing the student’s reliance on the teacher’s distribution as training advances, thereby freeing the student to develop independent reasoning skills. Following this phase, the framework utilizes self-knowledge distillation to serve as a structural regularizer. Using this method, we compressed the extensive Whisper architecture into a streamlined version known as ASKD-Whisper. Extensive testing across various acoustic environments reveals that ASKD-Whisper delivers a fivefold increase in inference speed while surpassing its teacher model with a 1.07% reduction in word error rate (WER). These findings confirm that ASKD successfully curbs overfitting induced by the teacher, setting a new benchmark for model compression that prioritizes generalizability.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC