Neural Architecture Search (NAS) is widely used to automatically obtain the neural network with the best performance among a large number of candidate architectures. To reduce the search time, zero-shot NAS aims at designing training-free proxies that can predict the test performance of a given architecture. However, as shown recently, none of the zero-shot proxies proposed to date can actually work consistently better than a naive proxy, namely, the number of network parameters (#Params). To improve this state of affairs, as the main theoretical contribution, we first reveal how some specific gradient properties across different samples impact the convergence rate and generalization capacity of neural networks. Based on this theoretical analysis, we propose a new zero-shot proxy, ZiCo, the first proxy that works consistently better than #Params. We demonstrate that ZiCo works better than State-Of-The-Art (SOTA) proxies on several popular NAS-Benchmarks (NASBench101, NATSBench-SSS/TSS, TransNASBench-101) for multiple applications (e.g., image classification/reconstruction and pixel-level prediction). Finally, we demonstrate that the optimal architectures found via ZiCo are as competitive as the ones found by one-shot and multi-shot NAS methods, but with much less search time. For example, ZiCo-based NAS can find optimal architectures with 78.1%, 79.4%, and 80.4% test accuracy under inference budgets of 450M, 600M, and 1000M FLOPs, respectively, on ImageNet within 0.4 GPU days. Our code is available at https://github.com/SLDGroup/ZiCo.

翻译：神经架构搜索（NAS）被广泛用于从大量候选架构中自动获得性能最优的神经网络。为减少搜索时间，零样本NAS旨在设计无需训练即可预测给定架构测试性能的代理指标。然而近期研究表明，迄今提出的所有零样本代理指标均未能持续优于一个朴素代理——即网络参数数量（#Params）。为改善这一现状，我们首先从理论层面揭示了不同样本间特定的梯度特性如何影响神经网络的收敛速度与泛化能力。基于该理论分析，我们提出新型零样本代理指标ZiCo，这是首个持续优于#Params的代理指标。在多个主流NAS基准（NASBench101、NATSBench-SSS/TSS、TransNASBench-101）上，我们证明ZiCo在图像分类/重建及像素级预测等多种应用场景中均优于现有最优（SOTA）代理指标。最后，通过ZiCo搜索得到的最优架构与单次/多次NAS方法所得结果竞争力相当，但搜索耗时大幅降低。例如，基于ZiCo的NAS方法在ImageNet上以450M、600M和1000M FLOPs推理预算下，分别仅需0.4 GPU天即可获得78.1%、79.4%和80.4%的测试准确率。我们的代码已发布于https://github.com/SLDGroup/ZiCo。