The public model zoo containing enormous powerful pretrained model families (e.g., ResNet/DeiT) has reached an unprecedented scope than ever, which significantly contributes to the success of deep learning. As each model family consists of pretrained models with diverse scales (e.g., DeiT-Ti/S/B), it naturally arises a fundamental question of how to efficiently assemble these readily available models in a family for dynamic accuracy-efficiency trade-offs at runtime. To this end, we present Stitchable Neural Networks (SN-Net), a novel scalable and efficient framework for model deployment. It cheaply produces numerous networks with different complexity and performance trade-offs given a family of pretrained neural networks, which we call anchors. Specifically, SN-Net splits the anchors across the blocks/layers and then stitches them together with simple stitching layers to map the activations from one anchor to another. With only a few epochs of training, SN-Net effectively interpolates between the performance of anchors with varying scales. At runtime, SN-Net can instantly adapt to dynamic resource constraints by switching the stitching positions. Extensive experiments on ImageNet classification demonstrate that SN-Net can obtain on-par or even better performance than many individually trained networks while supporting diverse deployment scenarios. For example, by stitching Swin Transformers, we challenge hundreds of models in Timm model zoo with a single network. We believe this new elastic model framework can serve as a strong baseline for further research in wider communities.

翻译：公开模型动物园中包含了前所未有的海量强大预训练模型族（例如ResNet/DeiT），这极大地推动了深度学习的发展。由于每个模型族包含不同规模的预训练模型（例如DeiT-Ti/S/B），自然引出一个基本问题：如何高效地整合同一模型族中这些现成的模型，以实现运行时动态的精度-效率权衡？为此，我们提出了可缝合神经网络（SN-Net），这是一种新颖的可扩展且高效的模型部署框架。给定一组预训练神经网络（称为锚点），该框架能以低成本生成大量具有不同复杂度和性能权衡的网络。具体而言，SN-Net在块/层处拆分锚点，然后通过简单的缝合层将它们连接起来，以将一个锚点的激活映射到另一个锚点。仅需少量训练周期，SN-Net即可有效插值具有不同规模的锚点性能。运行时，SN-Net通过切换缝合位置可即时适应动态资源约束。在ImageNet分类任务上的大量实验表明，SN-Net能够达到甚至超过许多单独训练网络的性能，同时支持多样的部署场景。例如，通过缝合Swin Transformer，我们仅用单个网络即可挑战Timm模型动物园中的数百个模型。我们相信这种新型弹性模型框架可为更广泛社区的研究提供强大基线。