Large Transformer models are capable of implementing a plethora of so-called in-context learning algorithms. These include gradient descent, classification, sequence completion, transformation, and improvement. In this work, we investigate whether large language models (LLMs), which never explicitly encountered the task of black-box optimization, are in principle capable of implementing evolutionary optimization algorithms. While previous works have solely focused on language-based task specification, we move forward and focus on the zero-shot application of LLMs to black-box optimization. We introduce a novel prompting strategy, consisting of least-to-most sorting of discretized population members and querying the LLM to propose an improvement to the mean statistic, i.e. perform a type of black-box recombination operation. Empirically, we find that our setup allows the user to obtain an LLM-based evolution strategy, which we call `EvoLLM', that robustly outperforms baseline algorithms such as random search and Gaussian Hill Climbing on synthetic BBOB functions as well as small neuroevolution tasks. Hence, LLMs can act as `plug-in' in-context recombination operators. We provide several comparative studies of the LLM's model size, prompt strategy, and context construction. Finally, we show that one can flexibly improve EvoLLM's performance by providing teacher algorithm information via instruction fine-tuning on previously collected teacher optimization trajectories.

翻译：大型Transformer模型能够实现多种所谓的上下文学习算法，包括梯度下降、分类、序列补全、变换和优化。本文研究从未明确接触过黑箱优化任务的大型语言模型（LLMs）是否原则上能够实现进化优化算法。不同于先前仅聚焦于基于语言的任务规范的研究，我们进一步关注将LLMs零样本应用于黑箱优化。我们提出一种新颖的提示策略，包括对离散化种群成员进行从少到多的排序，并查询LLM以提出对均值统计量的改进，即执行一种黑箱重组操作。实验发现，我们的设置允许用户获得一种基于LLM的进化策略，我们称之为`EvoLLM`，它在合成BBOB函数以及小型神经进化任务上稳健地优于随机搜索和高斯爬山法等基准算法。因此，LLMs可作为“即插即用”的上下文重组算子。我们提供了关于LLM模型大小、提示策略和上下文构建的多项比较研究。最后，我们表明，通过利用先前收集的教师优化轨迹进行指令微调来提供教师算法信息，可以灵活提升EvoLLM的性能。