Title: The Significance of Training Prompts: State-Adaptive Optimization for Resilient Fine-Tuning

Abstract:

Although prompt engineering is essential for unlocking the full potential of Large Language Models (LLMs) during inference, the function of prompts during the training phase has been largely overlooked. Current fine-tuning approaches generally regard training prompts as superficial variations, operating under the assumption that instructions with identical semantics produce the same learning results. We demonstrate, however, that this perceived equivalence is misleading. While paraphrased prompts may result in similar performance within a specific task, they trigger markedly divergent effects on cross-task capabilities, particularly concerning catastrophic forgetting and generalization. Importantly, we find that these cross-task effects are positively correlated, suggesting the presence of "superior" prompts that consistently enhance overall performance. Moreover, we identify that these optimal prompts can be reliably detected by analyzing task loss before the learning process begins. Drawing on these findings, we propose State-Adaptive Prompt Optimization (SAPO), a lightweight yet powerful training framework that transforms task formulation from a fixed input into a dynamic, state-dependent variable. Extensive experiments across various benchmarks validate SAPO’s efficacy, showing it significantly reduces forgetting and boosts generalization, thereby outperforming current state-of-the-art methods. These findings shed light on the influence of training prompts on learning dynamics and provide a practical methodology for robust fine-tuning. Our code is accessible at https://github.com/Eric8932/SAPO.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC