Title: Leveraging Reward Uncertainty to Foster Behavioral Diversity in Reinforcement Learning

Abstract:

Traditional reinforcement learning (RL) frameworks generally aim to identify a deterministic policy that maximizes the anticipated cumulative scalar reward. However, contemporary domains like scientific discovery and the fine-tuning of language models increasingly require diverse outcomes. Current solutions, such as entropy regularization or diversity bonuses, frequently necessitate delicate trade-offs that compromise performance for the sake of stochasticity or depend on heuristic metrics that may fail to align with optimal policy rankings. We posit that diversity is more effectively conceptualized as a rational reaction to uncertainty regarding rewards. When reward functions are imperfectly known—due to ambiguous human preferences or flawed reward models—adhering strictly to a single action choice often proves sub-optimal.

To address this, we introduce a fundamental restructuring of the RL objective. This approach substitutes the standard scalar reward with a probability distribution over multiple reward functions and employs a non-linear objective function across sets of actions. The resulting framework allows for the natural emergence of calibrated behavioral diversity. This diversity is manageable via the reward function distribution and is achieved without compromising expected reward performance. Within the context of contextual bandits, we develop a principled gradient estimator for this new objective and demonstrate that our formulation serves as a generalization of both standard policy gradient methods and more recent action-set techniques. Our empirical findings indicate that this framework provides a robust, theoretically sound alternative for complex RL tasks, particularly in scenarios where the conventional problem formulation fails to generate the necessary range of agent behaviors.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC