Title: ResMerge: Leveraging Residuals for Spectral Merging in Large Language Models

Abstract

While model merging provides a training-free mechanism for integrating multiple post-trained expert models, combining experts derived from reinforcement learning (RL) presents significant difficulties. Conventional spectral merging techniques typically operate on the premise that the primary task signal resides within the leading singular directions, allowing lower-energy residual components to be compressed, filtered, or attenuated to minimize interference. However, our analysis reveals that this assumption is invalid for RL task vectors. Upon decomposing each task vector into a dominant spectral head and a residual component, we observe that both segments are capable of independently restoring substantial behavioral knowledge, yet they display distinct merging characteristics. Specifically, the head component is highly concentrated and informative but susceptible to sharp conflicts between experts. In contrast, the residual component is more dispersed, offering a more stable foundation for aggregation.

Drawing on these insights, we introduce ResMerge, a residual-based spectral merging framework designed specifically for RL experts. The framework initially establishes a robust residual backbone via Spherical Residual Consensus Adaptation, a process that calculates a reliability-weighted consensus direction on the Frobenius sphere. Subsequently, it incorporates leading-head information through a Lightweight Head Correction module, which is activated by positive cross-expert agreement. Evaluations across various RL expert groups and capability domains demonstrate that ResMerge outperforms representative task-vector and spectral merging baselines in preserving expert capabilities. The code for ResMerge is publicly accessible at https://github.com/sunyd0303-cpu/ResMerge-release.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC