Title: LK Losses: Optimizing Direct Acceptance Rates for Speculative Decoding

Abstract:

Speculative decoding boosts the inference speed of autoregressive large language models (LLMs) by employing a lightweight draft model to suggest candidate tokens, which are subsequently verified in parallel by the target model. The magnitude of this speedup hinges heavily on the token acceptance rate. However, conventional training methods typically minimize Kullback-Leibler (KL) divergence as a surrogate objective. Although KL divergence and the acceptance rate share an identical global optimum, small draft models often possess limited capacity and consequently converge to suboptimal states. In these scenarios, reducing KL divergence fails to ensure the maximization of the acceptance rate.

To resolve this discrepancy, we introduce LK losses, a set of specialized training objectives designed to directly optimize for acceptance rate. Our comprehensive evaluation spans six target models (ranging from 8B to 685B parameters) and four distinct draft architectures. The results consistently show enhanced acceptance metrics across all configurations when compared to standard KL-based training. Furthermore, our approach demonstrates performance gains of up to 8-10% in average acceptance length across general, coding, and mathematical domains. LK losses are straightforward to implement, add no computational overhead, and can be seamlessly integrated into any existing speculative training framework, offering a robust alternative to current draft training objectives.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC