Title: PUMA: An Efficient Framework for Unified Multimodal Retrieval via Layer-Pruned Language Models and Modality-Adaptive Learning

Abstract:

The proliferation of multimedia data has driven a growing need for unified multimodal retrieval (UMR) systems in practical scenarios. While recent studies have adopted multimodal large language models (MLLMs) to address these requirements, their substantial parameter counts lead to prohibitive training expenses and sluggish inference speeds. To overcome these limitations, we introduce PUMA, a novel approach that integrates layer pruning with modality-adaptive learning to create an efficient unified multimodal retrieval system. Our methodology enhances UMR performance through both architectural and learning optimizations.

First, from an architectural standpoint, we develop Layer-Pruned Self-Distillation. This technique streamlines MLLMs by retaining only the shallow network layers, while leveraging features from the eliminated deeper layers as teacher signals for distillation. This strategy effectively lowers the parameter count without compromising the model’s representational power. Second, regarding the learning process, we propose the Modality-Adaptive Contrastive Learning Loss (MAC-Loss). This loss function categorizes in-batch negatives into distinct groups—harder intra-modality pairs and easier inter-modality pairs—based on the target modality. By applying tailored temperature strategies to these groups, MAC-Loss boosts learning efficiency. Empirical results demonstrate that our proposed method significantly cuts resource consumption while sustaining high performance levels.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC