Neural Architecture Search (NAS) benchmarks significantly improved the capability of developing and comparing NAS methods while at the same time drastically reduced the computational overhead by providing meta-information about thousands of trained neural networks. However, tabular benchmarks have several drawbacks that can hinder fair comparisons and provide unreliable results. These usually focus on providing a small pool of operations in heavily constrained search spaces -- usually cell-based neural networks with pre-defined outer-skeletons. In this work, we conducted an empirical analysis of the widely used NAS-Bench-101, NAS-Bench-201 and TransNAS-Bench-101 benchmarks in terms of their generability and how different operations influence the performance of the generated architectures. We found that only a subset of the operation pool is required to generate architectures close to the upper-bound of the performance range. Also, the performance distribution is negatively skewed, having a higher density of architectures in the upper-bound range. We consistently found convolution layers to have the highest impact on the architecture's performance, and that specific combination of operations favors top-scoring architectures. These findings shed insights on the correct evaluation and comparison of NAS methods using NAS benchmarks, showing that directly searching on NAS-Bench-201, ImageNet16-120 and TransNAS-Bench-101 produces more reliable results than searching only on CIFAR-10. Furthermore, with this work we provide suggestions for future benchmark evaluations and design. The code used to conduct the evaluations is available at https://github.com/VascoLopes/NAS-Benchmark-Evaluation.

翻译：神经架构搜索（NAS）基准通过提供数千个训练后神经网络的元信息，显著提升了NAS方法的开发与比较能力，同时大幅降低了计算开销。然而，表格型基准存在若干缺陷，可能阻碍公平比较并提供不可靠的结果。此类基准通常侧重于在高度受限的搜索空间中提供少量操作池——通常采用预定义外部骨架的基于细胞的神经网络。本文对广泛使用的NAS-Bench-101、NAS-Bench-201和TransNAS-Bench-101基准进行了实证分析，探究其泛化能力及不同操作对生成架构性能的影响。研究发现，仅需操作池中的部分子集即可生成接近性能上限的架构。此外，性能分布呈负偏态，即较高比例的架构集中于性能上限区间。我们一致发现卷积层对架构性能的影响最大，且特定的操作组合有利于高分架构。这些发现为使用NAS基准正确评估与比较NAS方法提供了见解，表明直接在NAS-Bench-201、ImageNet16-120和TransNAS-Bench-101上进行搜索比仅在CIFAR-10上搜索能产生更可靠的结果。此外，本研究为未来基准评估与设计提出了建议。用于执行评估的代码已发布于https://github.com/VascoLopes/NAS-Benchmark-Evaluation。