Benchmark Design in Black-Box Optimization (BBO) is a fundamental yet open-ended topic. Early BBO benchmarks are predominantly human-crafted, introducing expert bias and constraining diversity. Automating this design process can relieve the human-in-the-loop burden while enhancing diversity and objectivity. We propose Evolution of Benchmark (EoB), an automated BBO benchmark designer empowered by the large language model (LLM) and its program evolution capability. Specifically, we formulate benchmark design as a bi-objective optimization problem towards maximizing (i) landscape diversity and (ii) algorithm-differentiation ability across a portfolio of BBO solvers. Under this paradigm, EoB iteratively prompts LLM to evolve a population of benchmark programs and employs a reflection-based scheme to co-evolve the landscape and its corresponding program. Comprehensive experiments validate our EoB is a competitive candidate in multi-dimensional usages: 1) Benchmarking BBO algorithms; 2) Training and testing learning-assisted BBO algorithms; 3) Extending proxy for expensive real-world problems.

翻译：黑盒优化中的基准设计是一个基础性但开放的研究课题。早期的黑盒优化基准主要由人工构建，存在专家偏见且多样性受限。自动化该设计过程既能减轻人工参与负担，又能提升多样性与客观性。我们提出基准演化，一种由大语言模型及其程序演化能力驱动的自动化黑盒优化基准设计方法。具体而言，我们将基准设计建模为一个双目标优化问题，旨在最大化（i）景观多样性，以及（ii）在多种黑盒求解器上的算法区分能力。在此范式下，基准演化迭代式地提示大语言模型演化一组基准程序，并采用基于反思的机制协同演化景观及其对应程序。综合实验验证了我们的基准演化在多维应用场景中均具备竞争力：1）黑盒优化算法基准测试；2）学习辅助型黑盒优化算法的训练与测试；3）作为昂贵现实世界问题的替代代理。