Bayesian optimization is a coherent, ubiquitous approach to decision-making under uncertainty, with applications including multi-arm bandits, active learning, and black-box optimization. Bayesian optimization selects decisions (i.e. objective function queries) with maximal expected utility with respect to the posterior distribution of a Bayesian model, which quantifies reducible, epistemic uncertainty about query outcomes. In practice, subjectively implausible outcomes can occur regularly for two reasons: 1) model misspecification and 2) covariate shift. Conformal prediction is an uncertainty quantification method with coverage guarantees even for misspecified models and a simple mechanism to correct for covariate shift. We propose conformal Bayesian optimization, which directs queries towards regions of search space where the model predictions have guaranteed validity, and investigate its behavior on a suite of black-box optimization tasks and tabular ranking tasks. In many cases we find that query coverage can be significantly improved without harming sample-efficiency.

翻译：贝叶斯优化是一种在不确定性下进行决策的一致且广泛适用的方法，应用领域包括多臂老虎机、主动学习以及黑箱优化。贝叶斯优化通过最大化贝叶斯模型后验分布下的期望效用（该模型量化了查询结果中可约减的认知不确定性）来选取决策（即目标函数查询）。在实践中，由于以下两个原因，主观上不合理的查询结果可能频繁出现：1）模型错误设定；2）协变量偏移。共形预测是一种不确定性量化方法，即使对于错误设定的模型也能提供覆盖保证，并具有纠正协变量偏移的简单机制。我们提出共形贝叶斯优化，该方法将查询引导至模型预测具有保证有效性的搜索空间区域，并在多个黑箱优化任务和表格排序任务中研究了其行为。在许多情况下，我们发现查询覆盖性可在不损害样本效率的情况下得到显著提升。