Title: Vision Inference Former: Maintaining Visual Coherence in Multimodal Large Language Models

Abstract:

Multimodal Large Language Models (MLLMs) have recently seen significant advancements, largely driven by efficient methods for merging visual and textual data. The prevailing approach relies on a connector-based paradigm, which maps visual features into textual sequences to facilitate unified alignment and reasoning within a generative framework. However, our analysis highlights two critical shortcomings in this approach. First, despite visual data acting as the primary evidentiary source in MLLMs, it is processed with the same weight as text tokens, thereby undervaluing the distinct advantages of the visual modality. Second, as the length of the generated output increases—especially when constrained by a limited context window—the model’s reliance on visual cues gradually diminishes. This leads to a decline in vision-language alignment and a drop in the consistency between the generated text and the underlying visual semantics.

To overcome these issues, we introduce the Vision Inference Former (VIF), a compact architectural component designed to create a direct link between raw visual representations and the model’s output space. VIF works by continuously injecting visual semantics during the decoding stage of the inference process, ensuring that the model stays anchored to visual content throughout generation. We evaluated VIF across 14 benchmark tasks, including general reasoning, Optical Character Recognition (OCR), table comprehension, vision-centric assessments, and hallucination metrics. The results demonstrate that VIF reliably enhances performance across various architectures with negligible additional computational cost. The source code for this study is accessible at https://github.com/Dong-Xinpeng/VIF.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC