In this paper, we use Prior-data Fitted Networks (PFNs) as a flexible surrogate for Bayesian Optimization (BO). PFNs are neural processes that are trained to approximate the posterior predictive distribution (PPD) through in-context learning on any prior distribution that can be efficiently sampled from. We describe how this flexibility can be exploited for surrogate modeling in BO. We use PFNs to mimic a naive Gaussian process (GP), an advanced GP, and a Bayesian Neural Network (BNN). In addition, we show how to incorporate further information into the prior, such as allowing hints about the position of optima (user priors), ignoring irrelevant dimensions, and performing non-myopic BO by learning the acquisition function. The flexibility underlying these extensions opens up vast possibilities for using PFNs for BO. We demonstrate the usefulness of PFNs for BO in a large-scale evaluation on artificial GP samples and three different hyperparameter optimization testbeds: HPO-B, Bayesmark, and PD1. We publish code alongside trained models at https://github.com/automl/PFNs4BO.

翻译：本文利用先验数据拟合网络（PFNs）作为贝叶斯优化（BO）的灵活代理模型。PFNs是一种神经过程，通过从可高效采样的任意先验分布中进行上下文学习，训练其近似后验预测分布（PPD）。我们阐述了如何利用这种灵活性进行BO中的代理建模。我们使用PFNs分别模拟朴素高斯过程（GP）、高级高斯过程以及贝叶斯神经网络（BNN）。此外，我们还展示了如何将附加信息融入先验，例如允许关于最优点位置的提示（用户先验）、忽略无关维度，以及通过学习采集函数实现非贪婪贝叶斯优化。这些扩展所蕴含的灵活性为将PFNs应用于BO开辟了广阔的可能性。我们通过在人工GP样本和三个不同超参数优化测试平台（HPO-B、Bayesmark和PD1）上的大规模评估，证明了PFNs在BO中的实用性。我们已在https://github.com/automl/PFNs4BO上发布相关代码和训练模型。