Hyperparameter optimization (HPO) is concerned with the automated search for the most appropriate hyperparameter configuration (HPC) of a parameterized machine learning algorithm. A state-of-the-art HPO method is Hyperband, which, however, has its own parameters that influence its performance. One of these parameters, the maximal budget, is especially problematic: If chosen too small, the budget needs to be increased in hindsight and, as Hyperband is not incremental by design, the entire algorithm must be re-run. This is not only costly but also comes with a loss of valuable knowledge already accumulated. In this paper, we propose incremental variants of Hyperband that eliminate these drawbacks, and show that these variants satisfy theoretical guarantees qualitatively similar to those for the original Hyperband with the "right" budget. Moreover, we demonstrate their practical utility in experiments with benchmark data sets.

翻译：超参数优化（HPO）致力于自动搜索参数化机器学习算法的最优超参数配置。当前最先进的HPO方法是Hyperband，但该方法自身也包含影响其性能的参数。其中，最大预算这一参数尤为棘手：若设定过小，事后需要增加预算，而由于Hyperband并非增量式设计，整个算法必须重新执行。这不仅成本高昂，还会丢失已积累的宝贵知识。本文提出Hyperband的增量变体以消除这些缺陷，并证明这些变体在理论上与原版Hyperband在“恰当”预算下具有相当的质量保证。此外，我们通过基准数据集上的实验证明了其实际效用。