Title: Iterated Population Based Training with Task-Agnostic Restarts

Abstract: Hyperparameter Optimization (HPO) alleviates the manual effort required to tune neural network hyperparameters (HPs). Algorithms within the Population Based Training (PBT) family achieve efficiency by dynamically modifying HPs at intervals during weight optimization. However, recent findings highlight that the frequency of these HP updates serves as a critical meta-hyperparameter across all PBT variants, significantly influencing performance. Currently, there is no established intuition or method for efficiently determining this interval. To address this, we propose Iterated Population Based Training (IPBT), a new PBT variant that automatically tunes this parameter through task-agnostic restarts that preserve weight information. IPBT employs time-varying Bayesian optimization to reinitialize hyperparameters. Our evaluations across eight tasks involving image classification and reinforcement learning demonstrate that, on average, IPBT performs on par with or better than five prior PBT variants and other HPO methods, including random search, ASHA, and SMAC3. This superior performance is achieved without increasing the computational budget or altering any hyperparameters. The source code for IPBT is accessible at https://github.com/AwesomeLemon/IPBT.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC