Title: Coordinated Multi-Objective Alignment Spanning Verifiable and Subjective Rewards

Abstract: Although aligning large language models with human preferences is inherently a multidimensional challenge, most current training pipelines reduce these heterogeneous signals to a single objective. This study investigates the requirements for simultaneously aligning models across diverse domains, including those with verifiable rewards, non-verifiable subjective preferences, and complex interactive environments. Typically, such multi-objective configurations suffer from conflicting objectives that lead to inefficient training and restricted user control during inference. To overcome these limitations, we introduce $\textbf{M}$ulti-$\textbf{A}$ction-$\textbf{H}$ead $\textbf{AL}$ignment with PRM-guided Dec$\textbf{O}$ding ($\textbf{MAHALO}$). This unified framework standardizes Process Reward Model (PRM) training for step-level supervision in both verifiable and non-verifiable contexts. It executes vectorized multi-objective alignment via Multi-Action-Head DPO and facilitates controllable inference through objective-specific weighting and PRM-guided decoding. Evaluations in math reasoning, human value alignment, and multi-turn tutoring demonstrate that MAHALO enhances multiple objectives concurrently with minimal interference. The approach remains generalizable and adaptable across various domains while providing flexible user control at inference time. Our code is available at: https://github.com/pearls-lab/multiobj-align.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC