Title: Streamlining Hyperparameter Tuning for Large Language Model Reinforcement Learning

Abstract: Applying reinforcement learning (RL) to large language models (LLMs) is extremely sensitive to hyperparameter settings, rendering hyperparameter optimization (HPO) a critical but resource-heavy process. Current multi-fidelity HPO approaches struggle with efficiency in the context of LLM RL, largely because of the enormous scale of the models and the intensive nature of their training cycles. To address this, we introduce Joint Fidelity Hyperparameter Optimization (JF-HPO), a novel method that concurrently adjusts both the training budget and model size as fidelity metrics. JF-HPO is supported by three key innovations: (i) it utilizes a compact proxy model of the target LLM to facilitate rapid training and assessment during each HPO iteration; (ii) it employs strategically crafted early-stopping protocols grounded in training dynamics; and (iii) it incorporates a streamlined checkpointing system to remove unnecessary calculations. Relative to established HPO techniques, JF-HPO boosts the computational efficiency of individual trials by as much as 14.9 times, while maintaining predictive accuracy that is either superior or on par with existing methods within the same time constraints. Furthermore, when benchmarked against hyperparameter configurations derived from the VeRL Recipe, JF-HPO yields performance gains spanning from 5.8% to 111.6%.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC