The sample efficiency of Bayesian optimization algorithms depends on carefully crafted acquisition functions (AFs) guiding the sequential collection of function evaluations. The best-performing AF can vary significantly across optimization problems, often requiring ad-hoc and problem-specific choices. This work tackles the challenge of designing novel AFs that perform well across a variety of experimental settings. Based on FunSearch, a recent work using Large Language Models (LLMs) for discovery in mathematical sciences, we propose FunBO, an LLM-based method that can be used to learn new AFs written in computer code by leveraging access to a limited number of evaluations for a set of objective functions. We provide the analytic expression of all discovered AFs and evaluate them on various global optimization benchmarks and hyperparameter optimization tasks. We show how FunBO identifies AFs that generalize well in and out of the training distribution of functions, thus outperforming established general-purpose AFs and achieving competitive performance against AFs that are customized to specific function types and are learned via transfer-learning algorithms.

翻译：贝叶斯优化算法的采样效率取决于精心设计的采集函数，该函数指导函数评估的顺序收集。最佳采集函数在不同优化问题间差异显著，通常需要特定问题的手工选择。本研究致力于设计能在多种实验场景中表现优异的新型采集函数。基于FunSearch——一项利用大语言模型在数学科学领域进行发现的最新成果，我们提出FunBO：一种基于大语言模型的方法，可通过利用对目标函数集的有限评估次数，学习以计算机代码编写的新采集函数。我们提供了所有发现采集函数的解析表达式，并在多种全局优化基准测试和超参数优化任务中对其进行了评估。实验表明，FunBO能够识别在函数训练分布内外均具有良好泛化能力的采集函数，其性能超越成熟的通用采集函数，并与针对特定函数类型定制、通过迁移学习算法训练的采集函数达到相当水平。