Title: Unpacking Latent Visual Reasoning: A Mechanistic Diagnostic Approach

Abstract:

Recent advancements in latent visual reasoning have demonstrated significant performance improvements by integrating continuous latent tokens into multimodal language models. While these enhancements are typically credited to the tokens’ ability to encode visual evidence, recent investigations have uncovered a contradiction: the tokens appear only loosely connected to the image and provide minimal direct contribution to the final answer. A critical limitation of prior analyses is their treatment of latent tokens as an undifferentiated whole, which masks the actual drivers of these performance gains.

To address this, we decompose latent tokens into three distinct, testable elements: latent slots, boundary markers, and format. We employ a state-of-the-art method as a probe under optimal conditions to examine these components. Our findings, drawn from six method-stage configurations and four perception-intensive benchmarks, show that latent slots do not support the visual-memory hypothesis. In a striking reversal of expectations, retaining solely the boundary markers maintains between 78% and 100% of the performance gain across several settings. Furthermore, the model directs its attention more narrowly to the image at latent positions than at answer positions.

Consequently, the observed improvements stem from boundary markers, format, and this specific attention pattern, rather than from latent slots. The extent to which any given method leverages this mechanism is contingent upon its training supervision; notably, even when accuracy is matched, the underlying mechanisms can vary significantly. These results suggest that evaluating latent visual reasoning requires looking beyond mere accuracy metrics to understand what the model actually utilizes.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC