The rising growth of deep neural networks (DNNs) and datasets in size motivates the need for efficient solutions for simultaneous model selection and training. Many methods for hyperparameter optimization (HPO) of iterative learners, including DNNs, attempt to solve this problem by querying and learning a response surface while searching for the optimum of that surface. However, many of these methods make myopic queries, do not consider prior knowledge about the response structure, and/or perform a biased cost-aware search, all of which exacerbate identifying the best-performing model when a total cost budget is specified. This paper proposes a novel approach referred to as {\bf B}udget-{\bf A}ware {\bf P}lanning for {\bf I}terative Learners (BAPI) to solve HPO problems under a constrained cost budget. BAPI is an efficient non-myopic Bayesian optimization solution that accounts for the budget and leverages the prior knowledge about the objective function and cost function to select better configurations and to take more informed decisions during the evaluation (training). Experiments on diverse HPO benchmarks for iterative learners show that BAPI performs better than state-of-the-art baselines in most cases.

翻译：深度神经网络（DNN）及数据集的规模日益增长，催生了能够同时实现模型选择与训练的高效解决方案的需求。许多针对迭代学习器（包括DNN）的超参数优化（HPO）方法试图通过查询并学习响应曲面，同时搜索该曲面的最优解来解决这一问题。然而，这些方法中许多采用短视查询、未考虑响应结构的先验知识，和/或执行有偏的代价感知搜索，所有这些都加剧了在给定总成本预算时识别最佳模型的难度。本文提出一种名为面向迭代学习器的预算感知规划（BAPI）的新方法，用于解决在受限成本预算下的HPO问题。BAPI是一种高效的非短视贝叶斯优化方案，它兼顾预算约束，并利用目标函数与代价函数的先验知识来选取更优配置，并在评估（训练）过程中做出更明智的决策。在多个针对迭代学习器的多样化HPO基准测试上的实验表明，BAPI在大多数情况下优于当前最先进的基线方法。