This paper pursues the insight that language models naturally enable an intelligent variation operator similar in spirit to evolutionary crossover. In particular, language models of sufficient scale demonstrate in-context learning, i.e. they can learn from associations between a small number of input patterns to generate outputs incorporating such associations (also called few-shot prompting). This ability can be leveraged to form a simple but powerful variation operator, i.e. to prompt a language model with a few text-based genotypes (such as code, plain-text sentences, or equations), and to parse its corresponding output as those genotypes' offspring. The promise of such language model crossover (which is simple to implement and can leverage many different open-source language models) is that it enables a simple mechanism to evolve semantically-rich text representations (with few domain-specific tweaks), and naturally benefits from current progress in language models. Experiments in this paper highlight the versatility of language-model crossover, through evolving binary bit-strings, sentences, equations, text-to-image prompts, and Python code. The conclusion is that language model crossover is a promising method for evolving genomes representable as text.

翻译：本文围绕一个核心洞察展开：语言模型天然具备一种智能变异算子，其在本质上与进化算法中的交叉操作相似。具体而言，具备足够规模的语言模型展现出上下文学习能力——即通过少量输入模式间的关联进行学习，并生成包含此类关联的输出（亦称少量样本提示）。这一能力可被用于构建一种简洁而强大的变异算子：通过向语言模型提供若干基于文本的基因型（例如代码、自然语言句子或数学方程），并将其对应输出解析为这些基因型的后代。这种语言模型交叉方法（实现简单且可调用多种开源语言模型）的潜力在于：它提供了一种对语义丰富的文本表征进行进化的简易机制（仅需少量领域特异性调整），并能自然受益于语言模型领域的当前进展。本文通过进化二进制位串、句子、数学方程、文本到图像提示以及Python代码等案例，突出展示了语言模型交叉的普适性。结论表明，语言模型交叉是一种极具前景的、针对可表征为文本的基因组进行进化的方法。