Bayesian optimization (BO) is a powerful approach for optimizing complex and expensive-to-evaluate black-box functions. Its importance is underscored in many applications, notably including hyperparameter tuning, but its efficacy depends on efficiently balancing exploration and exploitation. While there has been substantial progress in BO methods, striking this balance still remains a delicate process. In this light, we present \texttt{LLAMBO}, a novel approach that integrates the capabilities of large language models (LLM) within BO. At a high level, we frame the BO problem in natural language terms, enabling LLMs to iteratively propose promising solutions conditioned on historical evaluations. More specifically, we explore how combining contextual understanding, few-shot learning proficiency, and domain knowledge of LLMs can enhance various components of model-based BO. Our findings illustrate that \texttt{LLAMBO} is effective at zero-shot warmstarting, and improves surrogate modeling and candidate sampling, especially in the early stages of search when observations are sparse. Our approach is performed in context and does not require LLM finetuning. Additionally, it is modular by design, allowing individual components to be integrated into existing BO frameworks, or function cohesively as an end-to-end method. We empirically validate \texttt{LLAMBO}'s efficacy on the problem of hyperparameter tuning, highlighting strong empirical performance across a range of diverse benchmarks, proprietary, and synthetic tasks.

翻译：贝叶斯优化（BO）是一种用于优化复杂且评估代价高昂的黑盒函数的强大方法。其重要性在诸多应用中得到凸显，尤其体现在超参数调优中，但其有效性依赖于高效平衡探索与利用。尽管贝叶斯优化方法已取得显著进展，但实现这种平衡仍是一个精细过程。基于此，我们提出\texttt{LLAMBO}——一种将大语言模型（LLM）能力融入贝叶斯优化的创新方法。在宏观层面，我们将贝叶斯优化问题以自然语言形式表述，使大语言模型能够基于历史评估结果迭代地提出有前景的候选解。具体而言，我们探索如何将大语言模型的上下文理解能力、少样本学习能力和领域知识相结合，以增强基于模型的贝叶斯优化的多个组件。实验结果表明，\texttt{LLAMBO}在零样本冷启动方面表现有效，并能改进代理建模和候选采样，尤其是在观测稀疏的搜索早期阶段。我们的方法以上下文方式执行，无需对大语言模型进行微调。此外，该方法具有模块化设计特性，既可将单个组件集成到现有贝叶斯优化框架中，也可作为端到端方法协同运行。我们通过超参数调优问题对\texttt{LLAMBO}的有效性进行了实证验证，结果表明该方法在多个不同基准测试、专有任务和合成任务上均展现出强劲的实证性能。