Recent work has demonstrated the promise of orchestrating large language models (LLMs) within evolutionary and agentic optimization systems. However, the mechanisms driving these optimization gains remain poorly understood. In this work, we present a large-scale study of LLM-guided evolutionary search, collecting optimization trajectories for 15 LLMs across 8 tasks. Although zero-shot problem-solving ability correlates with final optimization outcomes, it explains only part of the variance: models with similar initial capability often induce dramatically different search trajectories and outcomes. By analyzing these trajectories, we find that strong LLM optimizers behave as local refiners, producing frequent incremental improvements while progressively localizing the search in semantic space. Conversely, weaker optimizers exhibit large semantic drift, with sporadic breakthroughs followed by stagnation. Notably, various measures of solution novelty do not predict final performance; novelty is beneficial only when the search remains sufficiently localized around high-performing regions of the solution space. Our results highlight the importance of trajectory analysis for understanding and improving LLM-based optimization systems and provide actionable insights for their design and training.

翻译：近期研究表明，将大语言模型（LLM）融入演化与智能体优化系统具有显著潜力。然而，驱动这些优化增益的机制仍不明确。本文对LLM引导的演化搜索开展大规模研究，收集了15个LLM在8类任务中的优化轨迹。研究发现，零样本问题求解能力虽与最终优化结果相关，但仅能解释部分方差：初始能力相似的模型常引发截然不同的搜索轨迹与结果。通过轨迹分析发现，强LLM优化器表现为局部精炼器，在渐进式提升的同时逐步将搜索聚焦于语义空间中的局部区域；而弱优化器则呈现显著语义漂移，偶有突破但随即陷入停滞。值得注意的是，解的新颖性度量无法预测最终性能——新颖性仅在搜索持续集中于解空间高性能区域时才有益。本研究成果凸显了轨迹分析对理解与改进基于LLM的优化系统的重要性，并为系统设计与训练提供了可操作指导。