Hyperparameter optimization (HPO) is a vital step in improving performance in deep learning (DL). Practitioners are often faced with the trade-off between multiple criteria, such as accuracy and latency. Given the high computational needs of DL and the growing demand for efficient HPO, the acceleration of multi-objective (MO) optimization becomes ever more important. Despite the significant body of work on meta-learning for HPO, existing methods are inapplicable to MO tree-structured Parzen estimator (MO-TPE), a simple yet powerful MO-HPO algorithm. In this paper, we extend TPE's acquisition function to the meta-learning setting using a task similarity defined by the overlap of top domains between tasks. We also theoretically analyze and address the limitations of our task similarity. In the experiments, we demonstrate that our method speeds up MO-TPE on tabular HPO benchmarks and attains state-of-the-art performance. Our method was also validated externally by winning the AutoML 2022 competition on ``Multiobjective Hyperparameter Optimization for Transformers''.

翻译：超参数优化（HPO）是提升深度学习（DL）性能的关键步骤。实践者常面临多准则权衡问题，如精度与延迟的平衡。鉴于深度学习对计算资源的高需求以及高效HPO需求的日益增长，加速多目标（MO）优化变得愈发重要。尽管超参数优化领域的元学习研究已取得显著进展，现有方法无法适用于多目标树结构Parzen估计器（MO-TPE）——一种简洁而强大的MO-HPO算法。本文通过任务间顶级域重叠定义的任务相似性，将TPE的采集函数扩展至元学习场景。我们同时从理论上分析并解决了该任务相似性的局限性。实验表明，我们的方法在表格型HPO基准上加速了MO-TPE，并达到当前最优性能。该方法还在AutoML 2022"Transformer多目标超参数优化"竞赛中获胜，得到外部验证。