Bayesian optimization is a highly efficient approach to optimizing objective functions which are expensive to query. These objectives are typically represented by Gaussian process (GP) surrogate models which are easy to optimize and support exact inference. While standard GP surrogates have been well-established in Bayesian optimization, Bayesian neural networks (BNNs) have recently become practical function approximators, with many benefits over standard GPs such as the ability to naturally handle non-stationarity and learn representations for high-dimensional data. In this paper, we study BNNs as alternatives to standard GP surrogates for optimization. We consider a variety of approximate inference procedures for finite-width BNNs, including high-quality Hamiltonian Monte Carlo, low-cost stochastic MCMC, and heuristics such as deep ensembles. We also consider infinite-width BNNs and partially stochastic models such as deep kernel learning. We evaluate this collection of surrogate models on diverse problems with varying dimensionality, number of objectives, non-stationarity, and discrete and continuous inputs. We find: (i) the ranking of methods is highly problem dependent, suggesting the need for tailored inductive biases; (ii) HMC is the most successful approximate inference procedure for fully stochastic BNNs; (iii) full stochasticity may be unnecessary as deep kernel learning is relatively competitive; (iv) infinite-width BNNs are particularly promising, especially in high dimensions.

翻译：贝叶斯优化是一种高效优化查询代价高昂目标函数的方法。这类目标通常由高斯过程（GP）替代模型表示，其优化简便且支持精确推理。尽管标准GP替代模型已在贝叶斯优化中得到充分验证，但贝叶斯神经网络（BNN）近年来已成为实用的函数近似器，相较于标准GP具有诸多优势，例如能够自然处理非平稳性以及为高维数据学习表征。本文研究BNN作为标准GP替代模型在优化中的替代方案。我们考察了有限宽度BNN的多种近似推理方法，包括高质量哈密顿蒙特卡洛、低成本随机MCMC以及深度集成等启发式方法。同时考虑无限宽度BNN及部分随机化模型（如深度核学习）。我们在具有不同维度、目标数量、非平稳性以及离散与连续输入的问题上评估了这类替代模型集合。主要发现：（i）不同方法的排名高度依赖具体问题，表明需要定制化归纳偏置；（ii）HMC是完全随机化BNN中最成功的近似推理方法；（iii）完全随机化可能并非必需，因为深度核学习具有相对竞争力；（iv）无限宽度BNN尤其在处理高维问题时展现出显著潜力。