Title: KnapSpec: Enhancing Self-Speculative Decoding Through Adaptive Layer Selection Modeled as a Knapsack Problem

Abstract:

While Self-speculative decoding (SSD) significantly accelerates Large Language Model (LLM) inference by bypassing certain layers to generate an efficient draft model, current approaches frequently depend on static heuristics. These heuristics often fail to account for the dynamic computational overhead associated with attention mechanisms in long-context environments. To address this, we introduce KnapSpec, a training-free framework that reframes the selection of draft models as a knapsack problem, aiming to maximize tokens-per-time throughput. By separating Attention and MLP layers and characterizing their hardware-specific latencies as functions of context length, KnapSpec employs a parallel dynamic programming algorithm to dynamically identify optimal draft configurations in real-time. Additionally, we present the first rigorous theoretical analysis demonstrating that cosine similarity between hidden states serves as a mathematically robust proxy for token acceptance rates. This theoretical grounding enables our method to preserve high drafting faithfulness while adapting to the evolving bottlenecks of actual hardware. Evaluations on Qwen3 and Llama3 models show that KnapSpec consistently surpasses state-of-the-art SSD baselines, delivering wall-clock speedups of up to 1.47x across diverse benchmarks. As a plug-and-play solution, it facilitates high-speed inference for long sequences without necessitating additional training or altering the target model’s output distribution.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC