The wide application of pre-trained models is driving the trend of once-for-all training in one-shot neural architecture search (NAS). However, training within a huge sample space damages the performance of individual subnets and requires much computation to search for an optimal model. In this paper, we present PreNAS, a search-free NAS approach that accentuates target models in one-shot training. Specifically, the sample space is dramatically reduced in advance by a zero-cost selector, and weight-sharing one-shot training is performed on the preferred architectures to alleviate update conflicts. Extensive experiments have demonstrated that PreNAS consistently outperforms state-of-the-art one-shot NAS competitors for both Vision Transformer and convolutional architectures, and importantly, enables instant specialization with zero search cost. Our code is available at https://github.com/tinyvision/PreNAS.

翻译：预训练模型的广泛应用正推动一次性神经架构搜索（NAS）中“一次训练，处处适用”的趋势。然而，在庞大样本空间中进行训练会损害各个子网络的性能，且需要大量计算来搜索最优模型。本文提出PreNAS，一种在一次性训练中突出目标模型的无搜索NAS方法。具体而言，通过零成本选择器预先大幅缩减样本空间，并对优选架构进行权重共享的一次性训练以缓解更新冲突。大量实验表明，PreNAS在视觉Transformer与卷积架构上均始终优于最先进的一次性NAS方法，且关键在于实现了零搜索成本的即时特化。我们的代码开源在https://github.com/tinyvision/PreNAS。