Differentiable Architecture Search (DARTS) is an efficient Neural Architecture Search (NAS) method but suffers from robustness, generalization, and discrepancy issues. Many efforts have been made towards the performance collapse issue caused by skip dominance with various regularization techniques towards operation weights, path weights, noise injection, and super-network redesign. It had become questionable at a certain point if there could exist a better and more elegant way to retract the search to its intended goal -- NAS is a selection problem. In this paper, we undertake a simple but effective approach, named Smooth Activation DARTS (SA-DARTS), to overcome skip dominance and discretization discrepancy challenges. By leveraging a smooth activation function on architecture weights as an auxiliary loss, our SA-DARTS mitigates the unfair advantage of weight-free operations, converging to fanned-out architecture weight values, and can recover the search process from skip-dominance initialization. Through theoretical and empirical analysis, we demonstrate that the SA-DARTS can yield new state-of-the-art (SOTA) results on NAS-Bench-201, classification, and super-resolution. Further, we show that SA-DARTS can help improve the performance of SOTA models with fewer parameters, such as Information Multi-distillation Network on the super-resolution task.

翻译：可微分架构搜索（DARTS）是一种高效的神经架构搜索（NAS）方法，但存在鲁棒性、泛化性和差异性问题。针对由跳跃连接主导引起的性能崩溃问题，已有许多研究通过操作权重正则化、路径权重正则化、噪声注入和超网络重新设计等多种正则化技术进行改进。曾有一段时间，人们质疑是否存在一种更好、更优雅的方法，使搜索回归其本意——NAS是一个选择问题。本文采用一种简单而有效的方法，称为平滑激活DARTS（SA-DARTS），以克服跳跃连接主导和离散化差异的挑战。通过对架构权重施加平滑激活函数作为辅助损失，我们的SA-DARTS缓解了无权重操作的不公平优势，使架构权重值收敛至扇形分布，并能从跳跃连接主导的初始化中恢复搜索过程。通过理论和实证分析，我们证明SA-DARTS能在NAS-Bench-201、分类和超分辨率任务上取得新的最先进（SOTA）结果。此外，我们还表明SA-DARTS能以更少的参数帮助提升SOTA模型的性能，例如在超分辨率任务中的信息多重蒸馏网络。