One of the key challenges in Neural Architecture Search (NAS) is to efficiently rank the performances of architectures. The mainstream assessment of performance rankers uses ranking correlations (e.g., Kendall's tau), which pay equal attention to the whole space. However, the optimization goal of NAS is identifying top architectures while paying less attention on other architectures in the search space. In this paper, we show both empirically and theoretically that Normalized Discounted Cumulative Gain (NDCG) is a better metric for rankers. Subsequently, we propose a new algorithm, AceNAS, which directly optimizes NDCG with LambdaRank. It also leverages weak labels produced by weight-sharing NAS to pre-train the ranker, so as to further reduce search cost. Extensive experiments on 12 NAS benchmarks and a large-scale search space demonstrate that our approach consistently outperforms SOTA NAS methods, with up to 3.67% accuracy improvement and 8x reduction on search cost.

翻译：神经结构搜索(NAS)的主要挑战之一是对建筑的性能进行高效率的排序。对性能排级者进行主流评估时,使用对全空间同等重视的等级相关关系(例如Kendall's tau),然而,NAS的优化目标是确定顶级建筑,同时较少关注搜索空间中的其他建筑。在本文中,我们从经验上和理论上都表明,标准化的折扣累积增益(NDCG)是排级者更好的衡量标准。随后,我们提出了一个新的算法,AceNAS,直接优化与LamdaRank的NDCG。它还利用了由分享权重的NAS产生的弱标签来预先培养排级人员,以便进一步降低搜索成本。关于12个NAS基准和大型搜索空间的广泛实验表明,我们的做法始终超越SOTA NAS方法,精确度提高3.67%,搜索成本降低8x。