Neural Architecture Search (NAS) is widely used to automatically design the neural network with the best performance among a large number of candidate architectures. To reduce the search time, zero-shot NAS aims at designing training-free proxies that can predict the test performance of a given architecture. However, as shown recently, none of the zero-shot proxies proposed to date can actually work consistently better than a naive proxy, namely, the number of network parameters (#Params). To improve this state of affairs, as the main theoretical contribution, we first reveal how some specific gradient properties across different samples impact the convergence rate and generalization capacity of neural networks. Based on this theoretical analysis, we propose a new zero-shot proxy, ZiCo, the first proxy that works consistently better than #Params. We demonstrate that ZiCo works better than State-Of-The-Art (SOTA) proxies on several popular NAS-Benchmarks (NASBench101, NATSBench-SSS/TSS, TransNASBench-101) for multiple applications (e.g., image classification/reconstruction and pixel-level prediction). Finally, we demonstrate that the optimal architectures found via ZiCo are as competitive as the ones found by one-shot and multi-shot NAS methods, but with much less search time. For example, ZiCo-based NAS can find optimal architectures with 78.1%, 79.4%, and 80.4% test accuracy under inference budgets of 450M, 600M, and 1000M FLOPs on ImageNet within 0.4 GPU days.

翻译：神经架构搜索（NAS）广泛应用于从大量候选架构中自动设计性能最优的神经网络。为缩短搜索时间，零样本NAS致力于设计无需训练的代理指标，以预测给定架构的测试性能。然而近期研究表明，现有零样本代理指标均无法持续优于简单基线方法——即网络参数数量（#Params）。为改善这一现状，本文首先从理论层面揭示不同样本间特定梯度特性对神经网络收敛速度与泛化能力的影响机制。基于该理论分析，我们提出新型零样本代理指标ZiCo，这是首个可稳定优于#Params的代理指标。实验证明，ZiCo在多个主流NAS基准测试（NASBench101、NATSBench-SSS/TSS、TransNASBench-101）中均优于现有最优（SOTA）代理指标，可应用于图像分类/重建及像素级预测等任务。最终结果表明，通过ZiCo搜索获得的最优架构与单次/多次NAS方法具有同等竞争力，但搜索时间大幅缩短。例如，在ImageNet上以450M、600M和1000M FLOPs推理预算为约束条件，基于ZiCo的NAS仅需0.4 GPU天即可搜索获得测试准确率分别达78.1%、79.4%和80.4%的最优架构。