We use the Quality Diversity (QD) algorithm with Neural Cellular Automata (NCA) to automatically evaluate Multi-Agent Path Finding (MAPF) algorithms by generating diverse maps. Previously, researchers typically evaluate MAPF algorithms on a set of specific, human-designed maps at their initial stage of algorithm design. However, such fixed maps may not cover all scenarios, and algorithms may overfit to the small set of maps. To seek further improvements, systematic evaluations on a diverse suite of maps are needed. In this work, we propose Quality-Diversity Multi-Agent Path Finding Performance EvaluatoR (QD-MAPPER), a general framework that takes advantage of the QD algorithm to comprehensively understand the performance of MAPF algorithms by generating maps with patterns, be able to make fair comparisons between two MAPF algorithms, providing further information on the selection between two algorithms and on the design of the algorithms. Empirically, we employ this technique to evaluate and compare the behavior of different types of MAPF algorithms, including search-based, priority-based, rule-based, and learning-based algorithms. Through both single-algorithm experiments and comparisons between algorithms, researchers can identify patterns that each MAPF algorithm excels and detect disparities in runtime or success rates between different algorithms.

翻译：我们采用质量多样性算法与神经细胞自动机，通过生成多样化地图来自动评估多智能体路径规划算法。以往研究通常在算法设计初期使用一组特定的人工设计地图进行评估，然而此类固定地图可能无法涵盖所有场景，且算法容易在小规模地图集上过拟合。为寻求进一步改进，需要在多样化地图集合上进行系统性评估。本研究提出质量多样性多智能体路径规划性能评估框架，该通用框架利用质量多样性算法生成具有特定模式的地图，从而全面理解多智能体路径规划算法的性能表现，实现两种算法间的公平比较，并为算法选择与设计提供进一步依据。实证研究中，我们运用该技术评估比较了包括基于搜索、优先级、规则及学习等不同类型多智能体路径规划算法的行为特征。通过单算法实验与算法间对比，研究者可识别各多智能体路径规划算法擅长的模式，并检测不同算法在运行时间与成功率方面的差异。