Title: Semiparametric Preference Optimization: Your Language Model is Secretly a Single-Index Model

Abstract: Standard approaches to aligning policies with preference data generally rely on the assumption that a specific, known link function connects observed preferences to latent rewards, such as the logistic link used in the Bradley-Terry model. However, if this link function is misspecified, it can introduce bias into the inferred rewards and lead to misaligned policies. This paper investigates policy alignment when the link function is unknown and entirely unrestricted. We frame the problem as maximizing rewards under an $f$-divergence constraint and demonstrate that assuming the policy class is realizable leads to a semiparametric single-index binary choice model. In this framework, a scalar index generated by the policy encapsulates all dependencies on the demonstration data, while the rest of the preference distribution remains unconstrained.

Instead of following traditional econometric practices that seek to identify and estimate the structural parameters of such models, we propose methods that directly learn policies, treating the reward function as implicit. Our approach analyzes the error relative to the optimal policy and accommodates indices that are both nonparametric and unidentifiable. We establish convergence guarantees that are independent of the link function, expressed through generic measures of function complexity. Finally, we provide empirical validation for both our theoretical findings and proposed methods. The code is publicly accessible at https://github.com/causalml/spo/.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC