Title: Hint-Guided Diversified Policy Optimization for LLM Reasoning

Original: arXiv:2606.03021v1 Announce Type: new Abstract: Recent developments in Large Language Models (LLMs) have showcased impressive reasoning capabilities, with Reinforcement Learning with Verifiable Rewards (RLVR) being a promising enhancement strategy. However, existing reward mechanisms are constrained to the outcome-level correctness and lack explicit signals to guide the model to consider diverse solutions. In contrast, human problem solving typically involves evaluating multiple potential approaches and selecting the most reliable solution, a cognitive process that current RLVR frameworks do not explicitly incentivize. Inspired by this, we propose Hint-Guided Diversified Policy Optimization (HDPO), allowing the model to first list all potential candidate solution outlines as hints and then select the most reliable one for further reasoning. HDPO comprises two stages of Cold Start for Structured Reasoning and Hint-Guided Diversified Reinforcement Learning to incentivize the model to generate diverse and reliable solutions following the ``propose-select-think'' trajectory. Experimental results show that HDPO effectively boosts LLM reasoning and enhances the diversity of candidate solutions as well as the LLM's ability to identify reliable solutions.

Rewrite: Large Language Models (LLMs) have recently demonstrated significant reasoning prowess, with Reinforcement Learning with Verifiable Rewards (RLVR) emerging as a promising method for further improvement. Nevertheless, current reward systems are limited to verifying final outcomes and fail to provide explicit guidance for exploring varied solution paths. This stands in stark contrast to human cognition, where problem-solving generally entails assessing numerous potential strategies before choosing the most dependable one—a nuanced process that standard RLVR frameworks do not actively encourage. Drawing inspiration from this human-like approach, we introduce Hint-Guided Diversified Policy Optimization (HDPO). This method enables models to initially enumerate potential solution outlines as hints, subsequently selecting the most trustworthy path for deeper reasoning. HDPO operates through two distinct phases: Cold Start for Structured Reasoning and Hint-Guided Diversified Reinforcement Learning. These stages are designed to motivate the model to produce varied and dependable solutions by adhering to a “propose-select-think” workflow. Our experiments indicate that HDPO significantly improves LLM reasoning performance, while also increasing the variety of candidate solutions and the model’s proficiency in pinpointing reliable options.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC