Title: Follow-Your-Preference++: Reevaluating Preference Alignment Strategies in Image Inpainting

Abstract

This paper investigates the landscape of preference alignment within the domain of image inpainting. Instead of introducing a novel architectural method, we return to the fundamental principles of the problem to reassess its core difficulties. We utilize the established direct preference optimization framework, leveraging publicly accessible reward models to generate preference training data. Our comprehensive empirical analysis encompasses nine distinct reward models, two benchmark datasets, and two baseline inpainting models with varying architectures and generative mechanisms.

Our investigation yields several key insights: (1) While most reward models generate viable signals for constructing preference data, their reliability as evaluators varies significantly. (2) Preference data demonstrates consistent behavioral trends across different models and benchmarks, regardless of whether candidate or sample scaling is applied. (3) Reward models exhibit significant biases—specifically regarding brightness, composition, and color schemes—which increases the risk of reward hacking. (4) Implementing a straightforward ensemble of reward models effectively counteracts these biases, resulting in more robust and generalizable performance. (5) The methodology proves transferable to object removal tasks, shifting the objective from open-ended creative generation to the coherent completion of background elements. (6) Further examination indicates that a calibrated ensemble approach further reduces susceptibility to hacking while enhancing overall robustness.

Notably, without altering model architectures or incorporating additional datasets, our approach significantly surpasses previous state-of-the-art models. This superiority is confirmed through standard metrics, evaluations by large vision-language models, and human assessments. The source code for this work is publicly accessible at: https://github.com/shenytzzz/Follow-Your-Preference.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC