Bayesian optimization (BO) is a powerful approach for optimizing complex and expensive-to-evaluate black-box functions. Its importance is underscored in many applications, notably including hyperparameter tuning, but its efficacy depends on efficiently balancing exploration and exploitation. While there has been substantial progress in BO methods, striking this balance remains a delicate process. In this light, we present LLAMBO, a novel approach that integrates the capabilities of Large Language Models (LLM) within BO. At a high level, we frame the BO problem in natural language, enabling LLMs to iteratively propose and evaluate promising solutions conditioned on historical evaluations. More specifically, we explore how combining contextual understanding, few-shot learning proficiency, and domain knowledge of LLMs can improve model-based BO. Our findings illustrate that LLAMBO is effective at zero-shot warmstarting, and enhances surrogate modeling and candidate sampling, especially in the early stages of search when observations are sparse. Our approach is performed in context and does not require LLM finetuning. Additionally, it is modular by design, allowing individual components to be integrated into existing BO frameworks, or function cohesively as an end-to-end method. We empirically validate LLAMBO's efficacy on the problem of hyperparameter tuning, highlighting strong empirical performance across a range of diverse benchmarks, proprietary, and synthetic tasks.

翻译：贝叶斯优化（BO）是一种对复杂且评估代价高昂的黑箱函数进行优化的强大方法。其重要性在众多应用中得到凸显，尤其是在超参数调优领域，但其有效性取决于高效平衡探索与利用。尽管BO方法已取得显著进展，但仍难以精准维持这一平衡。为此，我们提出LLAMBO——一种将大型语言模型（LLM）能力融入贝叶斯优化的创新方法。宏观上，我们将BO问题转化为自然语言描述，使LLM能够基于历史评估结果迭代提出并评估有潜力的解。具体而言，我们探索了LLM的上下文理解、少样本学习能力及领域知识如何提升基于模型的贝叶斯优化。研究结果表明，LLAMBO在零样本冷启动、增强代理建模与候选采样方面成效显著，尤其在观测数据稀疏的搜索初期表现突出。该方法完全基于上下文运行，无需对LLM进行微调。此外，其模块化设计允许各组件独立集成至现有BO框架，亦可作为端到端方法协同运作。我们通过超参数调优任务验证了LLAMBO的有效性，在多样化基准测试、专有任务及合成任务中均展现出强劲的实证性能。