Title: Mechanistic Analysis and Spike-Free Quantization of Massive Spikes in LLMs as Bias Vectors

Abstract:

Large Language Models (LLMs) suffer significant quantization degradation due to massive activation spikes, which excessively expand dynamic ranges. Although previous theories have interpreted these anomalies as simple scalar biases, we contend that they are actually scalar intermediates derived from rigid, structural vector biases present in specific tokens. Our analysis demonstrates that, once normalized, these tokens converge toward constant vectors, thereby driving the attention sink and value-state drain mechanisms. We provide geometric evidence for this by examining the coordination of projection weights: $W_K$ serves to contrastively amplify the vector, $W_Q$ aligns semantic tokens with it, and $W_V$ projects it into the spectral null-space. Additionally, we find that the model actively protects these structural biases from Rotary Positional Embedding (RoPE) perturbations by confining them within "zones of rotational stability," achieved through the use of low-frequency bands and coherent channel pairs. Building on these insights, we introduce INSERTQUANT, a post-training quantization (PTQ) framework. This approach clamps spikes and restores their functional role using pre-computed template vectors, resulting in strictly spike-free activations that support robust, high-fidelity low-bit quantization. INSERTQUANT performs on par with leading per-tensor quantization methods for LLMs and offers the unique advantage of generalizing to other modalities, such as Vision Transformers (ViTs).

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC