This paper pursues the insight that language models naturally enable an intelligent variation operator similar in spirit to evolutionary crossover. In particular, language models of sufficient scale demonstrate in-context learning, i.e. they can learn from associations between a small number of input patterns to generate outputs incorporating such associations (also called few-shot prompting). This ability can be leveraged to form a simple but powerful variation operator, i.e. to prompt a language model with a few text-based genotypes (such as code, plain-text sentences, or equations), and to parse its corresponding output as those genotypes' offspring. The promise of such language model crossover (which is simple to implement and can leverage many different open-source language models) is that it enables a simple mechanism to evolve semantically-rich text representations (with few domain-specific tweaks), and naturally benefits from current progress in language models. Experiments in this paper highlight the versatility of language-model crossover, through evolving binary bit-strings, sentences, equations, text-to-image prompts, and Python code. The conclusion is that language model crossover is a promising method for evolving genomes representable as text.

翻译：本文探讨了语言模型天然具备的一种智能变异算子，其本质与进化算法中的交叉操作相似。具体而言，足够规模的语言模型展现出上下文学习能力——即能够从少量输入模式间的关联中学习，并生成包含此类关联的输出（亦称少样本提示）。这种能力可被用于构建一种简单而强大的变异算子：通过向语言模型提供少量基于文本的基因型（如代码、自然语句或数学表达式），并将其输出解析为这些基因型的后代。语言模型交叉（实现简单且可调用多种开源语言模型）的核心优势在于：能以极少领域特定调整，为语义丰富的文本表征提供进化机制，并自然受益于当前语言模型领域的进展。本文通过二进制比特串、自然语句、数学表达式、文本到图像提示及Python代码的进化实验，充分展示了语言模型交叉的普适性。结论表明，语言模型交叉是进化可表示为文本的基因型的极具前景的方法。