Recently, the weight-sharing technique has significantly speeded up the training and evaluation procedure of neural architecture search. However, most existing weight-sharing strategies are solely based on experience or observation, which makes the searching results lack interpretability and rationality. In addition, due to the negligence of fairness, current methods are prone to make misjudgments in module evaluation. To address these problems, we propose a novel neural architecture search algorithm based on dynamical isometry. We use the fix point analysis method in the mean field theory to analyze the dynamics behavior in the steady state random neural network, and how dynamic isometry guarantees the fairness of weight-sharing based NAS. Meanwhile, we prove that our module selection strategy is rigorous fair by estimating the generalization error of all modules with well-conditioned Jacobian. Extensive experiments show that, with the same size, the architecture searched by the proposed method can achieve state-of-the-art top-1 validation accuracy on ImageNet classification. In addition, we demonstrate that our method is able to achieve better and more stable training performance without loss of generality.

翻译：近年来，权重共享技术显著加速了神经架构搜索的训练与评估过程。然而，现有大多数权重共享策略仅基于经验或观察，导致搜索结果缺乏可解释性与合理性。此外，由于忽视公平性，当前方法在模块评估中易产生误判。为解决这些问题，我们提出一种基于动力等距的新型神经架构搜索算法。采用平均场理论中的不动点分析方法，探究稳态随机神经网络中的动力学行为，以及动力等距如何保障基于权重共享的NAS的公平性。同时，通过估计具有良态雅可比矩阵的所有模块的泛化误差，证明我们的模块选择策略具有严谨的公平性。大量实验表明，在相同规模下，本方法搜索得到的架构在ImageNet分类任务中能实现最先进的Top-1验证精度。此外，我们证明该方法在不损失泛化性的前提下，能够实现更优且更稳定的训练性能。