Title: Foundation-Preserving Adaptation via Generalized Rayleigh-Quotient Optimization

Abstract:

Although fine-tuning is a proven method for tailoring foundation models to specific downstream applications, it often compromises non-target competencies developed during the pretraining phase. Current approaches designed to mitigate catastrophic forgetting generally rely on specialized initialization techniques or static constraints to ensure safer parameter updates; however, these methods fail to actively manage the trade-off between preserving prior knowledge and adapting to new tasks throughout the training process.

To address this, we introduce Foundation Preserving LoRA (FoLoRA), an optimization framework that explicitly accounts for forgetting. FoLoRA operates under a first-order preservation condition, establishing a forgetting penalty based on activations derived from pretraining proxies and measuring task utility through downstream task activations. By calculating a generalized Rayleigh quotient—which evaluates task utility relative to the unit of forgetting penalty—the framework identifies optimal update directions. This process generates a spectral coordinate system that facilitates direction-wise gated Adam updates, effectively dampening updates that offer low utility relative to their penalty cost.

To accurately estimate the forgetting penalty, FoLoRA generates pretraining proxy calibration data by sampling directly from the pretrained model, thereby avoiding dependence on a single, potentially limited proxy dataset. Empirical evaluations across domains including mathematics, code generation, and instruction following demonstrate that FoLoRA outperforms baseline methods. It achieves a superior balance between preservation and adaptation, enhancing performance on target tasks while maintaining the highest aggregate retention of non-target capabilities.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC