Title: Pre-Fusion Refinement: Calibrating Multimodal Signals Through Contextual Awareness

Abstract:

While integrating data across linguistic, auditory, and visual channels typically enhances multimodal systems, this advantage is not universal. A specific modality may provide valuable insights for certain inputs while acting as noise for others, and internal feature responses within a single stream can contradict evidence found elsewhere. This study explores methods to refine multimodal representations prior to their integration by a downstream predictor. We introduce a lightweight calibration module that evaluates each modality against the others at a high level, identifying indicators of both agreement and disagreement among sources. These indicators are transformed into modulation signals that operate on both instance and dimension levels. By applying this calibration to raw modality features—rather than to already combined representations—the model can effectively dampen misleading elements, retain subtle yet relevant evidence, and amplify responses that align with the broader multimodal context. Functioning as a modular plug-in, this component can be integrated into various fusion backbones without necessitating changes to their prediction heads. Our approach demonstrates performance gains across five distinct benchmarks, including sentiment analysis, action recognition, audio-visual event detection, and emotion classification, under both convolutional and sequence-based fusion frameworks. Further investigations involving modality ablation, synthetic noise injection, training behavior, and feature visualization reveal that pre-fusion calibration mitigates interference from untrustworthy sources and fosters more stable multimodal optimization.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC