Title: When Attention Collapses: Stage-Aware Visual Token Pruning from Structure to Semantics

Original: arXiv:2606.03569v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) have demonstrated remarkable capabilities but suffer from significant computational overhead during inference. While visual token pruning offers a promising solution, existing methods predominantly rely on initial attention scores. This single-metric paradigm presents a critical flaw: high attention scores inherently collapse onto semantically similar regions, thereby severely reducing feature diversity and discarding vital contextual details. To address this, we introduce Structure-to-Semantics (STS), a novel two-stage visual token pruning framework that explicitly decouples the pruning process. The first stage employs a repulsion-based sampling mechanism to maximize spatial and structural diversity. The second stage leverages instruction-aware cross-attention to precisely filter out prompt-irrelevant tokens. This two-stage synergy constitutes the core of STS, first ensuring geometric coverage and then refining the retained tokens according to semantic relevance. Extensive evaluations demonstrate that STS mitigates the redundancy caused by attention-based selection, improving both structural diversity and fine-grained task alignment of the preserved visual tokens.

Rewrite: Although Vision-Language Models (VLMs) exhibit impressive performance, they incur substantial computational costs during the inference phase. Visual token pruning has emerged as a potential remedy; however, current approaches typically depend on a single metric: initial attention scores. This limitation creates a significant drawback, as high attention values tend to cluster around semantically comparable areas. Consequently, this convergence diminishes feature variety and eliminates essential contextual information. To overcome these challenges, we propose Structure-to-Semantics (STS), an innovative two-stage framework for visual token pruning that separates the selection process into distinct phases. In the initial stage, a repulsion-driven sampling technique is utilized to enhance spatial and structural variety. Subsequently, the second stage utilizes cross-attention mechanisms sensitive to instructions to accurately eliminate tokens that do not align with the prompt. The synergy between these two stages forms the foundation of STS, which first guarantees broad geometric coverage and subsequently optimizes the remaining tokens based on their semantic alignment. Comprehensive testing reveals that STS effectively reduces the redundancy associated with traditional attention-based selection, thereby enhancing both the structural diversity and the fine-grained task alignment of the kept visual tokens.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC