Title: Vegas: Verification-Guided Sparse Attention for Self-Speculative Decoding

Abstract:

In contemporary AI applications, long-context inference using large language models (LLMs) has become standard practice. However, this process is significantly hindered by the escalating memory requirements of the key-value (KV) cache. Prior research has demonstrated that self-speculative decoding combined with sparse attention can accelerate inference without compromising accuracy. This approach involves drafting tokens using a limited subset of the KV cache and verifying them in parallel against the complete cache. Yet, existing methods depend on separate KV selection algorithms to determine which entries are used for drafting, failing to recognize that the importance of each KV entry is naturally calculated during the verification phase.

To address this, we introduce Vegas, a self-speculative decoding framework that utilizes verification-guided sparse attention. Vegas leverages the verification process to identify critical KV cache entries as a secondary outcome, restricting attention computations to these vital entries when generating subsequent tokens. This strategy enhances the acceptance rate of draft tokens while maintaining low overhead for KV selection, ultimately boosting decoding throughput. Our evaluation shows that Vegas delivers a 1.25$\times$ to 2.81$\times$ increase in decoding throughput compared to the default vLLM implementation, and achieves a 1.15$\times$ to 1.29$\times$ improvement over leading sparse attention-based self-speculative decoding techniques. The source code is publicly accessible at https://github.com/platformxlab/vegas.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC