Title: Restoring MoE Functionality After Model Merging: A Training-Free Calibration Strategy

Original: arXiv:2606.03391v1 Announce Type: cross Abstract: Model merging has emerged as a cost-effective approach for consolidating the capabilities of multiple LLMs without retraining. However, existing merging techniques, largely based on linear parameter arithmetic or optimization, struggle when applied to Mixture-of-Experts (MoE) architectures. We identify a critical failure mode in MoE merging, termed routing breakdown, in which the merged router fails to dispatch tokens to suitable experts. Routing breakdown stems from the sensitivity of the non-linear softmax and discrete Top-k routing mechanisms to parameter perturbations from merging, a sensitivity further amplified by load-balancing constraints imposed during MoE pretraining. Because fine-tuned experts exhibit distinct specializations, even modest misrouting can cause severe performance degradation. To address this issue, we propose Hessian-Aware Router Calibration (HARC), a training-free framework that leverages second-order curvature information to realign the merged router. This approach admits a closed-form solution that can be efficiently solved using a matrix-free conjugate gradient method. Experiments on mathematical reasoning and code generation tasks show that HARC effectively mitigates routing breakdown across diverse MoE merging baselines and leads to substantial performance improvements. Our code is available at https://github.com/huangcb01/HARC.

Rewrite: Merging models has become an economical method for combining the strengths of various large language models (LLMs) without the need for retraining. Nevertheless, current merging strategies, which typically rely on linear parameter operations or optimization, encounter significant difficulties when used with Mixture-of-Experts (MoE) structures. This study highlights a specific failure point in MoE merging, referred to as routing breakdown, where the combined router is unable to direct tokens to the appropriate experts. This breakdown occurs because the non-linear softmax and discrete Top-k routing processes are highly sensitive to parameter changes introduced during merging. This sensitivity is heightened by load-balancing requirements established during the pretraining phase of MoE models. Since experts trained through fine-tuning develop unique specializations, even slight errors in routing can result in significant drops in performance. To overcome this challenge, we introduce Hessian-Aware Router Calibration (HARC), a training-free method that uses second-order curvature data to recalibrate the merged router. The method offers a closed-form solution, which can be computed efficiently via a matrix-free conjugate gradient algorithm. Tests involving code generation and mathematical reasoning demonstrate that HARC successfully resolves routing breakdown across various MoE merging baselines, resulting in notable performance gains. The code for this project can be found at https://github.com/huangcb01/HARC.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC