Title: Backward Transfer Optimization via Selective Refinement for Prompt-Based Continual Learning

Abstract: While prompt-based parameter-efficient continual learning helps mitigate catastrophic forgetting by isolating task-specific prompts, this isolation restricts the ability of later tasks to enhance earlier ones, leaving backward knowledge transfer largely underexplored. To address this limitation, we propose Selective bAckward refinement for positive Backward knowledge transfER (SABER), a replay-free framework designed to enable controlled backward transfer in prompt-based continual learning. SABER identifies when backward refinement is advantageous by employing complementary task-correlation criteria grounded in prompt-gradient geometry and loss-distribution similarity. Furthermore, it ensures safe refinement by constraining updates to non-interfering directions within the prompt parameter space. Extensive experiments across multiple continual learning benchmarks and diverse pretrained backbones, including T5-Large, LLaMA, and Qwen, demonstrate that SABER consistently achieves positive backward transfer while maintaining strong overall average performance. Code is available at https://github.com/OptMN-Lab/SABER-ICML-2026/.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC