Title: CRAM: Centroid-Routing and Adaptive MoE for Multimodal Continual Instruction Tuning

Abstract:

Multimodal Continual Instruction Tuning (MCIT) has become crucial for real-world applications, as Multimodal Large Language Models (MLLMs) must continuously expand their capabilities. While instruction tuning allows MLLMs to unify diverse vision-language tasks within a single generative framework, current approaches face a significant trade-off. Some methods update all tasks using a shared parameter set, which leads to catastrophic forgetting as heterogeneous tasks compete for the same resources. Others employ isolated modules for each new task, which avoids interference but results in poor parameter efficiency across extended sequences of tasks.

To resolve this conflict, we introduce CRAM. This approach isolates task-specific patterns into distinct modules to effectively mitigate catastrophic forgetting. Furthermore, CRAM enhances parameter efficiency through adaptive-rank instantiation. This mechanism identifies the gap between the capabilities of existing experts and the requirements of new tasks, allocating only the essential parameters needed. To facilitate stable knowledge reuse, centroid-guided routing activates the capabilities of existing experts, while an orthogonality penalty restricts new updates to task-specific directions, thereby preventing the re-learning of general capabilities. Comprehensive experiments on various benchmarks consistently show that CRAM outperforms existing methods.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC