Title: Deciphering Modality Interactions in Multimodal Language Models Through Partial Information Decomposition

Abstract:

Ensuring the reliable deployment of Multimodal Large Language Models (MLLMs) requires a deep understanding of how different modalities interact. To address this, we propose Partial Information Decomposition (PID) as a decision-level analytical framework. This approach goes beyond traditional representation alignment and outcome-based metrics by isolating the distinct, redundant, and synergistic contributions of sensory and linguistic inputs.

Our analysis of vision-language benchmarks uncovers consistent patterns in modality usage. Tasks focused on reasoning and grounding typically demonstrate high levels of synergy, while those requiring expert knowledge or factual recall rely more heavily on language-specific information. These usage profiles appear robust across various model architectures and can predict how models respond to interventions at the modality level.

We also adapt PID for tri-modal environments through "Sensory PID," which treats language as a control variable to break down information gain within video and audio streams. When applied to omni-modal models, Sensory PID identifies a bottleneck in sensory synergy that is primarily driven by visual data, even in tasks involving audio-visual fusion. Lastly, we provide preliminary evidence that reweighting strategies guided by PID can enhance performance in multimodal reasoning and grounding tasks.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC