Title: ProtoAda: Prototype-Guided Adaptive Adapter Expansion and Geometric Consolidation for Multimodal Continual Instruction Tuning

Abstract:

While Multimodal Large Language Models (MLLMs) demonstrate robust capabilities via instruction tuning, their practical application necessitates the continuous acquisition of new vision-language skills, rendering Multimodal Continual Instruction Tuning (MCIT) a critical requirement. To mitigate interference between tasks and foster collaborative learning, contemporary approaches frequently utilize sparse architectures, such as Mixture of LoRA Experts, which rely on image-text similarity for routing. However, this reliance on visual-linguistic similarity alone proves inadequate for accurate task allocation, as tasks with divergent response formats may still share similar semantic content, leading to incorrect routing. For instance, an expert trained on a grounding task requiring coordinate prediction might develop a bias toward generating concise textual responses after exposure to semantically analogous Visual Question Answering (VQA) tasks. This "format-blind" assignment forces heterogeneous output types into shared parameters, resulting in gradient interference and inefficient expert collaboration.

To resolve these challenges, we introduce ProtoAda, a framework for prototype-guided adaptive tuning. ProtoAda incorporates format-aware task prototypes to ensure that task assignment and routing are aligned with both the semantic content and the structural requirements of the output. Furthermore, it employs a geometry-aware strategy to consolidate updates that are compatible in format, thereby facilitating the effective reuse and progressive refinement of existing parameters. Comprehensive experiments across various benchmarks reveal that ProtoAda delivers superior performance, particularly excelling in scenarios where answer structures are prone to corruption during sequential tuning.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC