Title: Exploration Arises Naturally in Policy Gradient Reinforcement Learning Through Retry Mechanisms

Abstract: In the domain of reinforcement learning (RL), the utility of exploration stems from the agent’s ability to revisit similar states. The capacity to attempt diverse actions allows for performance enhancements and uncertainty reduction; conversely, in the absence of such opportunities for repetition, a greedy strategy proves optimal. We formalize this insight through ReMax, a novel objective function that assesses a policy based on the expected maximum return across $M$ samples ($M$ being a positive integer), while simultaneously factoring in return variability. By optimizing this objective, stochastic exploration emerges organically, eliminating the need for explicit bonus terms. To facilitate efficient policy optimization, we develop a new policy-gradient formulation for ReMax and propose ReMax PPO (RePPO). This variant of PPO optimizes the ReMax objective and extends the discrete retry count $M$ into a continuous parameter $m > 0$, thereby offering precise control over the degree of exploration. Experimental results on the MinAtar and Craftax benchmarks demonstrate that RePPO effectively encourages exploration without relying on any explicit exploration bonuses.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC