Title: MLLM-Microscope: Revealing the Latent Architecture of Multimodal Large Language Models

Abstract:

This study introduces MLLM-Microscope, an innovative framework engineered to dissect the concealed internal representations of Multimodal Large Language Models (MLLMs). The system is designed to assess key geometric properties—specifically linearity, intrinsic dimension, and anisotropy—of multimodal token embeddings as they traverse various transformer layers.

To demonstrate its utility, we applied MLLM-Microscope to two leading MLLM architectures, LLaVA-NeXT and OmniFusion, using the ScienceQA dataset. Our analysis reveals that token embeddings from both modalities, within both the primary and residual streams, display strongly linear characteristics across the model’s layers. However, distinct differences emerge between the two models: while OmniFusion maintains stable linearity, LLaVA-NeXT shows a marginal decrease in linearity for its image tokens. Furthermore, OmniFusion preserves a consistently higher intrinsic dimension for image tokens throughout all layers when compared to LLaVA-NeXT. Additionally, OmniFusion exhibits persistently low anisotropy across its layers.

These results indicate that the internal mechanics of MLLMs are heavily influenced by the specific modality fusion techniques employed prior to processing the token sequence within the LLM. The insights provided by MLLM-Microscope, along with other potential discoveries enabled by the system, offer valuable perspectives on MLLM operations. These findings have the potential to guide future advancements in model architecture design and optimization strategies.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC