Neural architecture search (NAS) has become a key component of AutoML and a standard tool to automate the design of deep neural networks. Recently, training-free NAS as an emerging paradigm has successfully reduced the search costs of standard training-based NAS by estimating the true architecture performance with only training-free metrics. Nevertheless, the estimation ability of these metrics typically varies across different tasks, making it challenging to achieve robust and consistently good search performance on diverse tasks with only a single training-free metric. Meanwhile, the estimation gap between training-free metrics and the true architecture performances limits training-free NAS to achieve superior performance. To address these challenges, we propose the robustifying and boosting training-free NAS (RoBoT) algorithm which (a) employs the optimized combination of existing training-free metrics explored from Bayesian optimization to develop a robust and consistently better-performing metric on diverse tasks, and (b) applies greedy search, i.e., the exploitation, on the newly developed metric to bridge the aforementioned gap and consequently to boost the search performance of standard training-free NAS further. Remarkably, the expected performance of our RoBoT can be theoretically guaranteed, which improves over the existing training-free NAS under mild conditions with additional interesting insights. Our extensive experiments on various NAS benchmark tasks yield substantial empirical evidence to support our theoretical results.

翻译：神经架构搜索（NAS）已成为AutoML的关键组成部分和自动化设计深度神经网络的标准工具。近年来，无训练NAS作为一种新兴范式，通过仅使用无训练指标评估真实架构性能，成功降低了基于标准训练方法NAS的搜索成本。然而，这些指标在不同任务上的估计能力通常存在差异，这使得仅使用单一无训练指标难以在多样化任务上实现鲁棒且持续优异的搜索性能。同时，无训练指标与真实架构性能之间的估计差距限制了无训练NAS实现更优性能。为解决这些挑战，我们提出了鲁棒化与增强的无训练NAS（RoBoT）算法，该算法：（a）利用贝叶斯优化探索得到的现有无训练指标的优化组合，构建一个在多样化任务上鲁棒且性能持续更优的指标；（b）对新生成的指标应用贪婪搜索（即利用策略），以弥合上述估计差距，从而进一步提升标准无训练NAS的搜索性能。值得注意的是，我们的RoBoT算法在理论上可保证期望性能，在温和条件下优于现有无训练NAS方法，并提供了额外的深刻洞见。我们在多个NAS基准任务上的大量实验为理论结果提供了充分的实证支持。