Title: Enhancing Large Language Model Distillation Through Concrete Score Matching

Abstract: Although Large Language Models (LLMs) exhibit exceptional performance, their high deployment costs have driven the adoption of knowledge distillation (KD) to facilitate more efficient inference. Traditional KD methods often rely on softmax-based probability matching between teacher and student models, a process that tends to obscure critical logit details. Although Direct Logit Distillation (DLD) addresses the issue of softmax smoothing, it overlooks the invariance of logits to shifts, which constrains the range of possible solutions. To address these limitations, we introduce Concrete Score Distillation (CSD), a discrete score-matching approach that eliminates both the blurring effect of softmax and the constraints on the optimal solution space. We demonstrate how to resolve the challenges of quadratic computational complexity and training instability associated with discrete score-matching in autoregressive LLMs. The resulting CSD objective flexibly weights and aligns relative logit differences across all vocabulary pairs between the student and teacher. Our framework includes both mode-seeking and mode-covering variants. We evaluated CSD on task-agnostic instruction-following and task-specific distillation benchmarks using GPT-2-1.5B, OpenLLaMA-7B, and GEMMA-7B-IT. Our experiments indicate that CSD consistently outperforms recent KD objectives, offers a strong balance between fidelity and diversity, and provides complementary benefits when integrated with on-policy methods, thereby proving its scalability and effectiveness for LLM distillation. Code: https://github.com/aailab-kaist/CSD.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC