The public model zoo containing enormous powerful pretrained model families (e.g., DeiT/Swin) has reached an unprecedented scope than ever, which significantly contributes to the success of deep learning. As each model family consists of pretrained models with diverse scales (e.g., DeiT-Ti/S/B), it naturally arises a fundamental question of how to effectively assemble these readily available models in a family for dynamic accuracy-efficiency trade-offs at runtime. In this work, we present Stitchable Neural Networks (SN-Net), a novel scalable and efficient framework for model deployment which cheaply produces numerous networks with different complexity and performance trade-offs. Specifically, SN-Net splits a family of pretrained neural networks, which we call anchors, across the blocks/layers and then stitches them together with simple stitching layers to map the activations from one anchor to another. With only a few epochs of training, SN-Net effectively interpolates between the performance of anchors with varying scales. At runtime, SN-Net can instantly adapt to dynamic resource constraints by switching the stitching positions. Furthermore, we provide a comprehensive study on what, how and where to stitch as well as a simple strategy for effectively and efficiently training SN-Net. Extensive experiments on ImageNet classification demonstrate that SN-Net can obtain on-par or even better performance than many individually trained networks while supporting diverse deployment scenarios. For example, by stitching Swin Transformers, we challenge hundreds of models in Timm model zoo with a single network. We believe this new elastic model framework can serve as a strong baseline for further research in wider communities.

翻译：公开模型库中包含了大量强大的预训练模型家族（例如DeiT/Swin），其规模达到了前所未有的程度，这对深度学习的成功做出了重要贡献。由于每个模型家族包含不同规模的预训练模型（例如DeiT-Ti/S/B），一个基本问题自然浮现：如何有效组合同一家族中这些现成的模型，以实现运行时动态的精度-效率权衡。在本文中，我们提出了可缝合神经网络（SN-Net），这是一种新颖的可扩展且高效的模型部署框架，能够廉价地生成大量具有不同复杂度和性能权衡的网络。具体而言，SN-Net将一组预训练神经网络（我们称之为锚点）按块或层进行拆分，然后通过简单的缝合层将它们连接起来，以将激活从一个锚点映射到另一个锚点。仅需少量训练周期，SN-Net就能有效插值出不同规模锚点之间的性能。在运行时，SN-Net可通过切换缝合位置立即适应动态资源约束。此外，我们提供了关于缝合什么、如何缝合以及何处缝合的全面研究，以及有效且高效训练SN-Net的简单策略。在ImageNet分类上的大量实验表明，SN-Net能够获得与许多单独训练网络相当甚至更好的性能，同时支持多样化的部署场景。例如，通过缝合Swin Transformer，我们仅用一个网络就挑战了Timm模型库中的数百个模型。我们相信，这种新的弹性模型框架可以成为更广泛社区进一步研究的强有力基准。