Hyperparameter optimization (HPO) is crucial for strong performance of deep learning algorithms and real-world applications often impose some constraints, such as memory usage, or latency on top of the performance requirement. In this work, we propose constrained TPE (c-TPE), an extension of the widely-used versatile Bayesian optimization method, tree-structured Parzen estimator (TPE), to handle these constraints. Our proposed extension goes beyond a simple combination of an existing acquisition function and the original TPE, and instead includes modifications that address issues that cause poor performance. We thoroughly analyze these modifications both empirically and theoretically, providing insights into how they effectively overcome these challenges. In the experiments, we demonstrate that c-TPE exhibits the best average rank performance among existing methods with statistical significance on 81 expensive HPO settings.

翻译：超参数优化对于深度学习算法取得优异性能至关重要，而实际应用通常在性能需求之上施加额外约束，如内存占用或延迟。本文提出约束TPE（c-TPE），这是对广泛使用的通用贝叶斯优化方法——树状Parzen估计器（TPE）的扩展，用于处理这些约束。我们所提出的扩展并非简单地将现有采集函数与原始TPE结合，而是包含针对导致性能不佳的缺陷所进行的改进。我们通过实验与理论分析深入探讨了这些改进，揭示了它们如何有效克服上述挑战。实验表明，在81个昂贵超参数优化场景中，c-TPE在现有方法中表现统计显著性最强的平均排名性能。