Title: Reward Score Matching: A Unified Framework for Fine-Tuning Flow and Diffusion Models

Abstract:

Reward-based fine-tuning aims to guide pretrained diffusion or flow-based generative models toward samples with higher reward values, while maintaining proximity to the original pretrained distribution. Although current methodologies emerge from varied theoretical standpoints, we demonstrate that many can be consolidated into a single unified framework, termed Reward Score Matching (RSM). Within this paradigm, alignment is conceptualized as score matching against a target conditioned on value guidance. Consequently, the primary distinctions among existing approaches lie in how they construct estimators for value guidance and how they modulate the optimization intensity across different timesteps.

This unified perspective elucidates the tradeoffs between bias, variance, and computational cost inherent in current designs, while also separating essential optimization components from auxiliary mechanisms that introduce complexity without providing distinct advantages. Leveraging these insights, we propose streamlined and more efficient redesigns for a range of representative reward alignment tasks, encompassing both differentiable and black-box settings. Ultimately, RSM transforms what appeared to be a disjointed array of fine-tuning techniques into a more compact, interpretable, and practical design space. The associated code can be accessed at https://github.com/jaylee2000/rsm.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC