Optimizing black-box functions is a fundamental problem in science and engineering. To solve this problem, many approaches learn a surrogate function that estimates the underlying objective from limited historical evaluations. Large Language Models (LLMs), with their strong pattern-matching capabilities via pretraining on vast amounts of data, stand out as a potential candidate for surrogate modeling. However, directly prompting a pretrained language model to produce predictions is not feasible in many scientific domains due to the scarcity of domain-specific data in the pretraining corpora and the challenges of articulating complex problems in natural language. In this work, we introduce LICO, a general-purpose model that extends arbitrary base LLMs for black-box optimization, with a particular application to the molecular domain. To achieve this, we equip the language model with a separate embedding layer and prediction layer, and train the model to perform in-context predictions on a diverse set of functions defined over the domain. Once trained, LICO can generalize to unseen molecule properties simply via in-context prompting. LICO achieves state-of-the-art performance on PMO, a challenging molecular optimization benchmark comprising over 20 objective functions.

翻译：黑箱函数优化是科学与工程领域的一个基础性问题。为解决该问题，许多方法通过学习一个代理函数，利用有限的历史评估数据来估计潜在目标。大语言模型凭借其通过海量数据预训练获得的强大模式匹配能力，成为代理建模的潜在候选方案。然而，在许多科学领域，由于预训练语料中领域特定数据的稀缺性以及用自然语言表述复杂问题的挑战，直接提示预训练语言模型进行预测并不可行。本工作提出了LICO，这是一个通用模型，可将任意基础大语言模型扩展用于黑箱优化，并特别应用于分子领域。为实现这一目标，我们为语言模型配备了独立的嵌入层和预测层，并训练该模型在领域内定义的各种函数上进行上下文预测。一旦训练完成，LICO仅需通过上下文提示即可泛化至未见过的分子性质。在PMO（一个包含超过20个目标函数的挑战性分子优化基准测试）上，LICO取得了最先进的性能。