Title: Abstraction-Driven Modality Alignment in Visual Instruction Tuning

Abstract:

Visual instruction tuning is a proven method for adapting pre-trained Large Language Models (LLMs) to interpret images in conjunction with text. However, the specific mechanism by which visual characteristics are integrated into the LLM backbone’s hierarchical abstraction structure has remained ambiguous. Through an examination of various vision-language architectures, our study reveals that instruction tuning functions primarily as a connector, routing visual features directly into the intermediate semantic layers of the LLM while circumventing the initial layers dedicated to unimodal processing. Utilizing causal interventions and probing analyses, we demonstrate that these mid-level layers constitute the semantic nucleus of vision-language tasks and are pivotal for achieving high performance across a wide range of multimodal benchmarks. Furthermore, an analysis of the geometric relationship between semantically equivalent visual and textual representations indicates that fine-tuning amplifies and reinforces the existing abstraction phase, thereby aligning visual inputs with established textual patterns. To validate the functional significance of this targeted alignment, we isolated fine-tuning to only the intermediate layers. This approach maintained the accuracy of full fine-tuning on vision-heavy benchmarks while significantly cutting down training duration. These findings indicate that multimodal integration is a localized process, driven by the repurposing of the LLM’s internal abstraction mechanisms.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC