Title: Dynamic Proxy-Mixing: Transferring Replay Controllers from Small to Large Models for Continual Instruction Tuning

Abstract:

Continual instruction tuning involves updating language models sequentially across new domains, a process that risks the gradual degradation of previously acquired capabilities and alignment behaviors. While replay serves as the primary countermeasure, fixed replay ratios are suboptimal because the ideal mixture depends on the current domain, the specific training phase, and the shifting vulnerability of prior knowledge. To address this, we introduce PROXYMIX, a framework that trains a dynamic replay controller on a small proxy model and then transfers this frozen controller to a larger target model. This controller operates without visibility into future tasks, deriving its state from normalized validation losses and their temporal trends to generate a masked mixture of the current task and available replay buffers.

We posit a core empirical hypothesis known as "forgetting mirroring," which suggests that rankings of task vulnerability remain consistent across different model scales, despite variations in absolute loss values. We empirically verify this assumption prior to executing the cross-scale transfer of controllers. In evaluations involving LLaMA-3-8B across five continual instruction tuning sequences, PROXYMIX outperforms the strongest non-oracle baseline, boosting average accuracy by 3.4 points, decreasing final forgetting by 3.5 points, and increasing safety scores by 5.8 points. Notably, this is achieved at approximately 50 times lower policy learning cost than Oracle Target RL. The framework is designed to be leakage-free and architecture-agnostic at the interface level. Furthermore, we delineate specific scenarios where the proxy assumption fails, thereby identifying constraints for robust deployment.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC