We present a novel application of evolutionary algorithms to automate the creation of powerful foundation models. While model merging has emerged as a promising approach for LLM development due to its cost-effectiveness, it currently relies on human intuition and domain knowledge, limiting its potential. Here, we propose an evolutionary approach that overcomes this limitation by automatically discovering effective combinations of diverse open-source models, harnessing their collective intelligence without requiring extensive additional training data or compute. Our approach operates in both parameter space and data flow space, allowing for optimization beyond just the weights of the individual models. This approach even facilitates cross-domain merging, generating models like a Japanese LLM with Math reasoning capabilities. Surprisingly, our Japanese Math LLM achieved state-of-the-art performance on a variety of established Japanese LLM benchmarks, even surpassing models with significantly more parameters, despite not being explicitly trained for such tasks. Furthermore, a culturally-aware Japanese VLM generated through our approach demonstrates its effectiveness in describing Japanese culture-specific content, outperforming previous Japanese VLMs. This work not only contributes new state-of-the-art models back to the open-source community, but also introduces a new paradigm for automated model composition, paving the way for exploring alternative, efficient approaches to foundation model development.

翻译：我们提出了一种新颖的进化算法应用，用于自动化构建强大的基础模型。尽管模型合并因其成本效益已成为大语言模型开发中颇具前景的方法，但当前该方法仍依赖人类直觉和领域知识，限制了其潜力。在此，我们提出一种进化方法，通过自动发现多样化开源模型的有效组合来克服这一局限，无需大量额外训练数据或计算资源即可利用其集体智慧。该方法同时作用于参数空间和数据流空间，从而实现对单个模型权重之外的优化。该方案甚至支持跨领域合并，生成如具备数学推理能力的日语大语言模型等混合模型。令人惊喜的是，我们生成的日语数学大语言模型在多个权威日语大语言模型基准测试中达到了最先进水平，甚至超越了参数量显著更大的模型，尽管其并未针对此类任务进行显式训练。此外，通过该方法生成的文化感知日语视觉语言模型在描述日本文化特定内容方面展现出卓越效能，性能超越了此前所有日语视觉语言模型。本工作不仅为开源社区贡献了新型最先进模型，更开创了自动化模型组合的新范式，为基础模型开发的替代高效路径探索奠定了基础。