Title: SSSD: Simply-Scalable Speculative Decoding

Original: arXiv:2411.05894v3 Announce Type: replace-cross

Abstract:

While speculative decoding has gained traction as a strategy to speed up Large Language Model (LLM) inference, its impact on production serving systems is often limited to marginal gains. Although some techniques deliver significant performance boosts, they generally depend on extra trained draft models or auxiliary components, which complicates both deployment and ongoing maintenance. This added intricacy hampers adaptability, especially when the workload transitions to tasks, domains, or languages that the draft model’s training data does not adequately cover.

To address these challenges, we present Simply-Scalable Speculative Decoding (SSSD), a novel approach that requires no training. SSSD integrates lightweight n-gram matching with hardware-aware speculation. Compared to conventional autoregressive decoding, this method can cut latency by as much as 2.9x. It matches the performance of top-tier training-dependent methods across numerous benchmarks, yet it demands far less effort to implement—eliminating the need for data preparation, training, or tuning. Furthermore, SSSD demonstrates greater resilience to shifts in language and domain, as well as improved stability in long-context scenarios.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC