The wide application of pre-trained models is driving the trend of once-for-all training in one-shot neural architecture search (NAS). However, training within a huge sample space damages the performance of individual subnets and requires much computation to search for an optimal model. In this paper, we present PreNAS, a search-free NAS approach that accentuates target models in one-shot training. Specifically, the sample space is dramatically reduced in advance by a zero-cost selector, and weight-sharing one-shot training is performed on the preferred architectures to alleviate update conflicts. Extensive experiments have demonstrated that PreNAS consistently outperforms state-of-the-art one-shot NAS competitors for both Vision Transformer and convolutional architectures, and importantly, enables instant specialization with zero search cost. Our code is available at https://github.com/tinyvision/PreNAS.

翻译：预训练模型的广泛应用推动了一次性神经架构搜索中“一次训练、处处可用”的趋势。然而，在巨大的样本空间中训练会损害单个子网络的性能，且需要大量计算来搜索最优模型。本文提出PreNAS，一种无搜索的神经架构搜索方法，其在一次性训练中突出目标模型。具体而言，通过零成本选择器提前大幅缩减样本空间，并在偏好架构上执行权重共享的一次性训练以缓解更新冲突。大量实验表明，PreNAS在视觉Transformer和卷积架构上均始终优于最先进的一次性NAS方法，且重要的是，能以零搜索成本实现即时特化。我们的代码已开源在https://github.com/tinyvision/PreNAS。