Title: c-TPE: Incorporating Inequality Constraints into Tree-Structured Parzen Estimators for Expensive Hyperparameter Optimization

Abstract: Hyperparameter optimization (HPO) plays a pivotal role in achieving robust performance in deep learning models. However, practical deployments frequently introduce additional restrictions, such as limits on memory consumption or inference latency, alongside standard performance metrics. To address this, we introduce constrained TPE (c-TPE), a novel adaptation of the popular Bayesian optimization technique known as the tree-structured Parzen estimator (TPE), specifically designed to manage such constraints. This approach does not merely layer an existing acquisition function onto the original TPE framework; rather, it incorporates targeted structural modifications to resolve specific factors that typically lead to suboptimal results. We provide comprehensive theoretical and empirical analyses of these adjustments, elucidating their mechanisms for effectively mitigating these challenges. Our experimental evaluation reveals that c-TPE achieves statistically significant superior average rank performance compared to current methods across 81 expensive HPO tasks involving inequality constraints. While the applicability of our method to hard-constrained optimization is explored in Appendix D due to the scarcity of comparable baselines, the code is now accessible through OptunaHub.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC