Reading, Not Thinking: Understanding and Bridging the Modality Gap When Text Becomes Pixels in Multimodal LLMs

Abstract

While Multimodal Large Language Models (MLLMs) are capable of processing text embedded within images, their performance frequently lags behind that observed when identical content is delivered as standard textual tokens. This study offers a comprehensive diagnosis of this "modality gap" by assessing seven distinct MLLMs across five different input modes and seven benchmarks. The evaluation covers a wide spectrum of sources, ranging from synthetically generated text to authentic document images extracted from Wikipedia pages and arXiv PDFs.

Our findings reveal that this performance disparity is highly dependent on specific rendering parameters, such as font style and resolution. Notably, natural document images demonstrate significantly smaller gaps, indicating that the observed performance drop may stem partly from evaluation artifacts rather than inherent model limitations. Through a grounded-theory error analysis of more than 4,000 instances, we pinpoint the root cause: providing input solely as an image suppresses the model’s reasoning efforts. Specifically, models generate outputs that are 5 to 19 times shorter, often bypassing step-by-step calculations or logical deduction.

The data suggests that the reluctance to engage in reasoning, rather than deficiencies in perception or knowledge retrieval, is the primary driver of the performance gap, especially for tasks demanding multi-step logic. To address this, we introduce a lightweight, on-policy self-distillation technique. By fine-tuning models using their own reasoning traces from text-mode tasks paired with image inputs, we successfully bridge this gap. This approach boosts image-mode accuracy by over 50%, allowing it to match or surpass text-mode performance. Furthermore, these improvements generalize to unseen benchmarks without causing catastrophic forgetting. Collectively, our results offer a systematic understanding of the modality gap and propose a viable strategy for enhancing visual text comprehension in multimodal language models.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC