Albeit being a prevalent architecture searching approach, differentiable architecture search (DARTS) is largely hindered by its substantial memory cost since the entire supernet resides in the memory. This is where the single-path DARTS comes in, which only chooses a single-path submodel at each step. While being memory-friendly, it also comes with low computational costs. Nonetheless, we discover a critical issue of single-path DARTS that has not been primarily noticed. Namely, it also suffers from severe performance collapse since too many parameter-free operations like skip connections are derived, just like DARTS does. In this paper, we propose a new algorithm called RObustifying Memory-Efficient NAS (ROME) to give a cure. First, we disentangle the topology search from the operation search to make searching and evaluation consistent. We then adopt Gumbel-Top2 reparameterization and gradient accumulation to robustify the unwieldy bi-level optimization. We verify ROME extensively across 15 benchmarks to demonstrate its effectiveness and robustness.

翻译：可微分架构搜索（DARTS）虽是一种流行的架构搜索方法，但由于整个超网络驻留内存导致其面临显著的内存成本制约。单路径DARTS应运而生，该方法在每步仅选择单一路径子模型。这种方案在兼顾内存友好性的同时，也具备低计算成本的优势。然而我们发现单路径DARTS存在一个尚未被充分关注的关键问题：与DARTS类似，由于派生过多跳跃连接类无参数操作，其同样面临严重的性能崩溃。本文提出名为RObustifying Memory-Efficient NAS（ROME）的新算法以解决该问题。首先，我们将拓扑搜索与操作搜索解耦以确保搜索与评估的一致性。进而采用Gumbel-Top2重参数化与梯度累积机制来优化棘手的双层优化过程。我们在15个基准测试上对ROME进行广泛验证，证明其有效性与鲁棒性。