Recently, the weight-sharing technique has significantly speeded up the training and evaluation procedure of neural architecture search. However, most existing weight-sharing strategies are solely based on experience or observation, which makes the searching results lack interpretability and rationality. In addition, due to the negligence of fairness, current methods are prone to make misjudgments in module evaluation. To address these problems, we propose a novel neural architecture search algorithm based on dynamical isometry. We use the fix point analysis method in the mean field theory to analyze the dynamics behavior in the steady state random neural network, and how dynamic isometry guarantees the fairness of weight-sharing based NAS. Meanwhile, we prove that our module selection strategy is rigorous fair by estimating the generalization error of all modules with well-conditioned Jacobian. Extensive experiments show that, with the same size, the architecture searched by the proposed method can achieve state-of-the-art top-1 validation accuracy on ImageNet classification. In addition, we demonstrate that our method is able to achieve better and more stable training performance without loss of generality.

翻译：最近，权重共享技术显著加速了神经架构搜索的训练与评估过程。然而，现有大多数权重共享策略仅基于经验或观察，导致搜索结果缺乏可解释性与合理性。此外，由于忽视公平性，当前方法在模块评估中易产生误判。为解决这些问题，我们提出一种基于动态等距的新型神经架构搜索算法。利用平均场理论中的定点分析方法，我们分析了稳态随机神经网络的动力学行为，以及动态等距如何保障基于权重共享的NAS的公平性。同时，通过估计所有模块在雅可比条件良好情况下的泛化误差，我们证明了模块选择策略的严格公平性。大量实验表明，在相同规模下，本方法搜索得到的架构在ImageNet分类任务中可实现最优的top-1验证准确率。此外，我们证明所提方法能在不损失泛化能力的前提下，实现更优且更稳定的训练性能。