Recent work pairs LLMs with evolutionary search to iteratively generate, modify, and select code using task-specific feedback. These systems have produced strong results in mathematical discovery and algorithm design, yet a fundamental question remains: what do they actually evolve? Progress is typically summarized by the best score a run reaches under a task-specific evaluator, but that score can reflect several different mechanisms: new algorithmic structure, re-tuning an existing strategy, recombining ideas already in the model's internal knowledge, or overfitting to the evaluator. Distinguishing these mechanisms requires inspecting the search process itself, not only its final outcome. We introduce EvoTrace, a dataset of evolutionary coding traces spanning four evolutionary frameworks, reasoning and non-reasoning models, and 16 tasks across mathematics and algorithm design. To analyze these traces, we develop EvoReplay, a replay-based methodology that reconstructs the local search states behind high-scoring solutions and tests controlled interventions, including adjusting constants, removing program components and substituting models or prompting contexts. We annotate every code edit in EvoTrace with one of nine recurring edit types using an LLM-as-judge pipeline validated against blind human re-annotation. Across EvoTrace, most score gains come from a small subset of these edit types. We further find a deterministic cycling pattern: about 30% of code lines added during search are byte-identical re-introductions of previously-deleted lines, present throughout nearly every run. These results show that benchmark gains in evolutionary coding agents can arise from qualitatively different mechanisms, only some of which correspond to new algorithmic structure. EvoTrace enables more diagnostic evaluation of evolutionary coding agents beyond final benchmark scores.

翻译：近期的研究将大语言模型与进化搜索相结合，利用任务特定反馈迭代式地生成、修改和选择代码。这类系统在数学发现和算法设计方面取得了显著成果，但一个根本性问题依然存在：它们究竟在进化什么？通常，进展通过任务特定评估器下运行达到的最佳得分来总结，但该得分可能反映多种不同机制：新的算法结构、对现有策略的重新调参、重组模型内部知识中已有的思路，或对评估器过度拟合。区分这些机制需要审视搜索过程本身，而不仅仅是最终结果。我们引入了EvoTrace，一个跨越四种进化框架、推理与非推理模型以及数学与算法设计领域16个任务的进化编码轨迹数据集。为分析这些轨迹，我们开发了基于回放的方法论EvoReplay，它重构高分解决方案背后的局部搜索状态，并测试受控干预，包括调整常量、移除程序组件以及替换模型或提示上下文。我们利用经过盲人人工重标注验证的LLM-as-judge流水线，将EvoTrace中的每次代码编辑标注为九种重复编辑类型之一。在EvoTrace中，大多数得分提升来自这些编辑类型中的一小部分。我们还发现一种确定性循环模式：搜索过程中添加的代码行中，约30%是之前删除行的字节相同再引入，这几乎出现在每一次运行中。这些结果表明，进化编码代理的基准测试得分提升可能源于性质不同的机制，其中只有部分对应新的算法结构。EvoTrace使得对进化编码代理的诊断性评估超越了最终的基准测试得分。