Title: PRISM: A Preference-Aware Influence Function Approach for Efficient Fine-Tuning Data Selection

Abstract

As Large Language Models (LLMs) continue to expand in scale, enhancing training efficiency has become increasingly dependent on the effective utilization of data. Data selection addresses this challenge by directing limited training resources toward high-value examples that best promote the model’s desired behavior. Current methodologies typically define target behavior through a collection of reference examples, scoring candidate training data based on their estimated influence on these samples. However, these conventional approaches treat all target examples with equal significance, overlooking the fact that individual examples vary in their relevance to model optimization. Specifically, examples that closely align with the model’s inherent behavior provide strong supervisory signals, while those that diverge offer only weak and ineffective local guidance.

To address this, we introduce PRISM (Preference-aware Influence function based Data Selection Method). PRISM utilizes model preferences to assign weights to target examples, thereby constructing a preference-aware target direction. It then evaluates candidate training samples based on their influence on this direction, ensuring that the data budget is prioritized for samples that effectively steer the model toward the expected target behavior. Our theoretical analysis confirms that this weighted preference construction yields a superior first-order gradient direction for enhancing target preference, outperforming uniform aggregation strategies. Extensive experiments across various model architectures and parameter scales demonstrate that PRISM delivers improved performance in both efficient fine-tuning and safety-aligned supervised fine-tuning rectification. These results underscore that accurately characterizing target behavior is fundamental to cost-effective data selection.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC