Hyperparameter optimization (HPO) is concerned with the automated search for the most appropriate hyperparameter configuration (HPC) of a parameterized machine learning algorithm. A state-of-the-art HPO method is Hyperband, which, however, has its own parameters that influence its performance. One of these parameters, the maximal budget, is especially problematic: If chosen too small, the budget needs to be increased in hindsight and, as Hyperband is not incremental by design, the entire algorithm must be re-run. This is not only costly but also comes with a loss of valuable knowledge already accumulated. In this paper, we propose incremental variants of Hyperband that eliminate these drawbacks, and show that these variants satisfy theoretical guarantees qualitatively similar to those for the original Hyperband with the "right" budget. Moreover, we demonstrate their practical utility in experiments with benchmark data sets.

翻译：超参数优化（HPO）关注于自动化搜索参数化机器学习算法的最优超参数配置（HPC）。当前最先进的HPO方法是Hyperband，但其自身参数会影响性能。其中一个关键参数——最大预算，尤为棘手：若取值过小，事后需增加预算，而由于Hyperband并非增量式设计，必须重新运行整个算法。这不仅成本高昂，还会丢失已积累的宝贵知识。本文提出Hyperband的增量变体以消除这些缺陷，并证明这些变体在理论上具有与原始Hyperband在"正确"预算下本质相似的保证。此外，我们通过基准数据集实验验证了其实际效用。