Title: SENSE: Semantic Embedding Navigation with Soft-gated Evaluation for Retrieval-based Speculative Decoding

Abstract

Speculative Decoding (SD) enhances the efficiency of Large Language Model (LLM) inference by utilizing a lightweight draft model to generate candidate tokens. These candidates are then verified in parallel by the target model, a process that maintains generation quality while accelerating throughput. Although Retrieval-based Speculative Decoding (RSD) is widely appreciated for its modular, plug-and-play nature, its effectiveness is often limited by strict lexical dependencies. This rigidity makes both the retrieval and verification stages sensitive to superficial textual variations.

To overcome these limitations, we introduce SENSE (Semantic Embedding Navigation with Soft-gated Evaluation). SENSE anchors the retrieval process to the hidden states of the target model, thereby establishing robust semantic alignment. This approach enables the Soft-gated Evaluation module to verify semantic equivalence rather than relying on exact surface forms. To facilitate rigorous and granular, component-level comparisons, we deconstruct existing methods into atomic primitives within a unified framework. Extensive experiments across various domains show that SENSE surpasses several baseline methods in the LLaMA and Qwen model families, achieving a mean acceptance length of up to 4.09 and a speedup of 3.26x, all while preserving generation quality. The source code will be made publicly available upon publication.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC