Automatically optimizing the hyperparameters of Machine Learning algorithms is one of the primary open questions in AI. Existing work in Hyperparameter Optimization (HPO) trains surrogate models for approximating the response surface of hyperparameters as a regression task. In contrast, we hypothesize that the optimal strategy for training surrogates is to preserve the ranks of the performances of hyperparameter configurations as a Learning to Rank problem. As a result, we present a novel method that meta-learns neural network surrogates optimized for ranking the configurations' performances while modeling their uncertainty via ensembling. In a large-scale experimental protocol comprising 12 baselines, 16 HPO search spaces and 86 datasets/tasks, we demonstrate that our method achieves new state-of-the-art results in HPO.

翻译：自动优化机器学习算法的超参数是人工智能领域未解决的主要开放问题之一。现有超参数优化（HPO）工作通过训练代理模型将超参数响应曲面近似为回归任务。相比之下，我们假设训练代理模型的最优策略是将超参数配置的性能排名保持为"学习排序"问题。为此，我们提出一种新方法，通过元学习优化排序配置性能的神经网络代理模型，并利用集成方法对其不确定性进行建模。在包含12个基线方法、16个HPO搜索空间和86个数据集/任务的大规模实验方案中，我们证明该方法在HPO领域取得了新的最优结果。