Weight-sharing neural architecture search aims to optimize a configurable neural network model (supernet) for a variety of deployment scenarios across many devices with different resource constraints. Existing approaches use evolutionary search to extract models of different sizes from a supernet trained on a very large data set, and then fine-tune the extracted models on the typically small, real-world data set of interest. The computational cost of training thus grows linearly with the number of different model deployment scenarios. Hence, we propose Transfer-Once-For-All (TOFA) for supernet-style training on small data sets with constant computational training cost over any number of edge deployment scenarios. Given a task, TOFA obtains custom neural networks, both the topology and the weights, optimized for any number of edge deployment scenarios. To overcome the challenges arising from small data, TOFA utilizes a unified semi-supervised training loss to simultaneously train all subnets within the supernet, coupled with on-the-fly architecture selection at deployment time.

翻译：权重共享的神经架构搜索旨在优化一个可配置的神经网络模型（超网络），使其适用于众多具有不同资源约束的设备上的多种部署场景。现有方法采用进化搜索从在大规模数据集上训练的超网络中提取不同规模的模型，然后针对通常规模较小的实际目标数据集进行微调。因此，训练的计算成本随不同模型部署场景的数量线性增长。为此，我们提出一次性转移适用于所有方法（TOFA），用于在小数据集上进行超网络式训练，且训练计算成本不随边缘部署场景数量增加而变化。对于给定任务，TOFA能够为任意数量的边缘部署场景获取定制化的神经网络，包括拓扑结构和权重。为克服小数据带来的挑战，TOFA采用统一的半监督训练损失，同时训练超网络中的所有子网络，并在部署时动态进行架构选择。