Title: Saliency-Aware Model Merging

Abstract:

The goal of model merging is to integrate several task-specific models, each fine-tuned on distinct datasets, into a single architecture capable of demonstrating proficiency across multiple domains. However, existing data-free merging techniques frequently face scalability challenges. These methods typically depend on rudimentary parameter-level heuristics, which overlook the complex inter-layer dependencies and the uneven distribution of expertise among models.

To address these limitations, this study introduces SA-Merging, a framework grounded in connectivity-based saliency formulations derived from structural pruning techniques such as SynFlow, adapted for the data-free model merging context. We establish a saliency metric for task vectors in relation to a common base model. Additionally, we propose a merge-aware modulation mechanism that leverages consensus among experts to reduce task interference. Through this formulation, an iterative merging process gradually eliminates non-informative updates while maintaining end-to-end connectivity.

Moreover, we adapt SA-Merging to facilitate rank-wise saliency decomposition for LoRAs, ensuring their structural integrity remains intact. Comprehensive experiments across both vision and language tasks validate the efficacy of our saliency-driven approach, significantly narrowing the performance disparity between data-free methods and those utilizing test-time adaptation.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC