Bayesian optimization (BO), while proved highly effective for many black-box function optimization tasks, requires practitioners to carefully select priors that well model their functions of interest. Rather than specifying by hand, researchers have investigated transfer learning based methods to automatically learn the priors, e.g. multi-task BO (Swersky et al., 2013), few-shot BO (Wistuba and Grabocka, 2021) and HyperBO (Wang et al., 2022). However, those prior learning methods typically assume that the input domains are the same for all tasks, weakening their ability to use observations on functions with different domains or generalize the learned priors to BO on different search spaces. In this work, we present HyperBO+: a pre-training approach for hierarchical Gaussian processes that enables the same prior to work universally for Bayesian optimization on functions with different domains. We propose a two-step pre-training method and analyze its appealing asymptotic properties and benefits to BO both theoretically and empirically. On real-world hyperparameter tuning tasks that involve multiple search spaces, we demonstrate that HyperBO+ is able to generalize to unseen search spaces and achieves lower regrets than competitive baselines.

翻译：贝叶斯优化（Bayesian optimization, BO）虽被证明对许多黑箱函数优化任务高度有效，但需要实践者精心选择能良好建模目标函数的先验。为替代人工指定，研究者已探索基于迁移学习的自动先验学习方法，例如多任务BO（Swersky等，2013）、小样本BO（Wistuba和Grabocka，2021）及HyperBO（Wang等，2022）。然而，这些先验学习方法通常假设所有任务的输入域相同，这削弱了其利用不同域函数观测数据的能力，或难以将学习到的先验泛化至不同搜索空间的BO问题。本文提出HyperBO+：一种面向层次高斯过程的预训练方法，使得同一先验能通用地适用于不同域函数的贝叶斯优化。我们提出两步式预训练方法，并从理论与实证角度分析其理想的渐近性质及对BO的益处。在涉及多搜索空间的真实超参数调优任务中，我们证明HyperBO+能泛化至未见搜索空间，且比竞争基线取得更低的累积遗憾。