Recently, the Deep Learning community has become interested in evolutionary optimization (EO) as a means to address hard optimization problems, e.g. meta-learning through long inner loop unrolls or optimizing non-differentiable operators. One core reason for this trend has been the recent innovation in hardware acceleration and compatible software - making distributed population evaluations much easier than before. Unlike for gradient descent-based methods though, there is a lack of hyperparameter understanding and best practices for EO - arguably due to severely less 'graduate student descent' and benchmarking being performed for EO methods. Additionally, classical benchmarks from the evolutionary community provide few practical insights for Deep Learning applications. This poses challenges for newcomers to hardware-accelerated EO and hinders significant adoption. Hence, we establish a new benchmark of EO methods (NeuroEvoBench) tailored toward Deep Learning applications and exhaustively evaluate traditional and meta-learned EO. We investigate core scientific questions including resource allocation, fitness shaping, normalization, regularization & scalability of EO. The benchmark is open-sourced at https://github.com/neuroevobench/neuroevobench under Apache-2.0 license.

翻译：近期，深度学习领域开始关注进化优化（Evolutionary Optimization, EO）方法，以解决硬优化问题，例如通过长内部循环展开的元学习或优化非可微算子。这一趋势的核心原因在于硬件加速与兼容软件的最新创新，使得分布式种群评估比以往更加便捷。然而，与基于梯度下降的方法不同，EO方法缺乏超参数理解和最佳实践——这很大程度上归因于针对EO方法开展的"研究生下降"（graduate student descent）和基准测试严重不足。此外，进化计算领域的经典基准测试对深度学习应用的实践指导意义有限。这给硬件加速EO领域的新手带来挑战，并阻碍了该方法的广泛采用。为此，我们建立了一个面向深度学习应用的EO方法新基准（NeuroEvoBench），并全面评估了传统EO与元学习EO。我们探究了核心科学问题，包括资源分配、适应度塑形、归一化、正则化及EO的可扩展性。该基准在https://github.com/neuroevobench/neuroevobench以Apache-2.0许可证开源发布。