Title: A Local Perturbation Theory for Cross-Domain Interference and Recovery in Multi-Domain RL

Original: arXiv:2606.02398v1 Announce Type: new Abstract: Post-training reinforcement learning (RL) enhances large language models (LLMs) in specific areas like mathematical reasoning, code generation, question answering, and creative writing (CW). However, focusing on one domain frequently leads to a decline in performance across other domains. Current theories attributing this to catastrophic forgetting or global gradient conflicts fail to provide a complete picture, as significant interference persists even when full-model gradients are almost orthogonal. Our analysis reveals that single-domain RL generates sparse, low-magnitude parameter updates with minimal overlap among the most significantly altered neurons. Despite this sparsity, distinct domains continue to utilize substantial shared computational pathways, where the direction of updates dictates whether they cooperate or clash. Leveraging this insight, we establish a theoretical framework based on local perturbations in multi-domain RL, demonstrating that subsequent domain training primarily damages earlier domains through a second-order term. This damage is concentrated within a low-dimensional conflict subspace, a consequence of the observed sparse route structure. Furthermore, we find that a brief domain refresh can shrink this harmful component within the subspace, allowing for selective recovery with minimal side effects. Aligning with our theoretical predictions, a short Re-Math refresh following a Code → Math → QA → CW sequence improved Math performance from 57.66 to 66.04, while largely maintaining scores in other areas and achieving a top average score of 66.39. Additionally, a training-free rollback targeting a sparse set of proxy conflict coordinates for the Math-QA pair partially restored Math capabilities, offering direct evidence at the proxy level that damage is indeed localized. These findings offer a mechanistic understanding of how interference and recovery operate in multi-domain RL through a localized lens.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC