Title: TaDA: Calibrated Probe Gating for Task-Domain LoRA Merging

Abstract

Merging a task-specific LoRA adapter with a domain-specific LoRA adapter into a cohesive model presents a practical challenge that remains largely underexplored. Current approaches typically view these adapters as equivalent entities, distributing uniform weights across every layer. However, we contend that task and domain adapters display a consistent asymmetry dependent on depth within transformer architectures. Specifically, domain influence intensifies as layer depth increases, whereas shallower layers preserve more robust task-relevant information.

Driven by this insight, we introduce TaDA (Task-Domain LoR$\textbf{A}$ Merging), a training-free algorithm designed to leverage this structural hierarchy. TaDA employs calibrated probe-guided gating on a per-layer basis and performs subspace-aware merging for individual components. This gating mechanism assigns distinct weights to each layer and projection type based on a probe signal that has been proven invariant to the magnitude of adapter weights. Furthermore, the merging process eliminates conflicting singular directions prior to combining the remaining elements. The result is a standard rank-$r$ LoRA adapter that incurs zero inference overhead.

In evaluations across six scientific question-answering benchmarks using Llama-2-7B, TaDA attained an average accuracy of 0.452. This performance surpasses DARE-TIES by 3.6 percentage points and secured the top spot on all six benchmarks. Additionally, on six image classification benchmarks utilizing ViT-L/16, TaDA achieved an average accuracy of 85.9%. This represents an improvement over the most effective merging baseline and led in three out of the six individual benchmarks.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC