Bayesian optimization (BO) has traditionally solved black-box problems where function evaluation is expensive and, therefore, observations are few. Recently, however, there has been growing interest in applying BO to problems where function evaluation is cheaper and observations are more plentiful. In this regime, scaling to many observations $N$ is impeded by Gaussian-process (GP) surrogates: GP hyperparameter fitting scales as $\mathcal{O}(N^3)$ (reduced to roughly $\mathcal{O}(N^2)$ in modern implementations), and it is repeated at every BO iteration. Many methods improve scaling at acquisition time, but hyperparameter fitting still scales poorly, making it the bottleneck. We propose Epistemic Nearest Neighbors (ENN), a lightweight alternative to GPs that estimates function values and uncertainty (epistemic and aleatoric) from $K$-nearest-neighbor observations. ENN scales as $\mathcal{O}(N)$ for both fitting and acquisition. Our BO method, TuRBO-ENN, replaces the GP surrogate in TuRBO with ENN and its Thompson-sampling acquisition with $\mathrm{UCB} = μ(x) + σ(x)$. For the special case of noise-free problems, we can omit fitting altogether by replacing $\mathrm{UCB}$ with a non-dominated sort over $μ(x)$ and $σ(x)$. We show empirically that TuRBO-ENN reduces proposal time (i.e., fitting time + acquisition time) by one to two orders of magnitude compared to TuRBO at up to 50,000 observations.

翻译：贝叶斯优化（BO）传统上用于解决黑箱问题，其中函数评估成本高昂，因此观测数据稀少。然而，近期越来越多的研究关注将BO应用于函数评估成本较低、观测数据更为丰富的问题。在此场景下，高斯过程（GP）代理模型阻碍了算法向大规模观测数据（$N$）的扩展：GP超参数拟合的计算复杂度为$\mathcal{O}(N^3)$（在现代实现中通常降至约$\mathcal{O}(N^2)$），且该过程在每次BO迭代中均需重复执行。尽管许多方法改进了采集阶段的计算效率，但超参数拟合的复杂度依然居高不下，成为性能瓶颈。本文提出认知最近邻（ENN），一种轻量级的GP替代方案，通过$K$最近邻观测数据估计函数值及其不确定性（认知不确定性与偶然不确定性）。ENN在拟合与采集阶段的计算复杂度均为$\mathcal{O}(N)$。我们的BO方法TuRBO-ENN将TuRBO中的GP代理模型替换为ENN，并将其汤普森采样采集函数替换为$\mathrm{UCB} = μ(x) + σ(x)$。针对无噪声问题的特殊情形，我们可通过采用基于$μ(x)$和$σ(x)$的非支配排序替代$\mathrm{UCB}$，完全省略拟合过程。实验表明，在高达50,000次观测的规模下，相较于TuRBO，TuRBO-ENN将提案时间（即拟合时间与采集时间之和）降低了一至两个数量级。