This paper introduces TinySaver, an early-exit-like dynamic model compression approach which employs tiny models to substitute large models adaptively. Distinct from traditional compression techniques, dynamic methods like TinySaver can leverage the difficulty differences to allow certain inputs to complete their inference processes early, thereby conserving computational resources. Most existing early exit designs are implemented by attaching additional network branches to the model's backbone. Our study, however, reveals that completely independent tiny models can replace a substantial portion of the larger models' job with minimal impact on performance. Employing them as the first exit can remarkably enhance computational efficiency. By searching and employing the most appropriate tiny model as the computational saver for a given large model, the proposed approaches work as a novel and generic method to model compression. This finding will help the research community in exploring new compression methods to address the escalating computational demands posed by rapidly evolving AI models. Our evaluation of this approach in ImageNet-1k classification demonstrates its potential to reduce the number of compute operations by up to 90%, with only negligible losses in performance, across various modern vision models. The code of this work will be available.

翻译：本文提出TinySaver，一种类似早期退出的动态模型压缩方法，通过自适应地使用小模型替代大模型。与传统的压缩技术不同，像TinySaver这样的动态方法可以充分利用难度差异，允许某些输入提前完成推理过程，从而节省计算资源。现有的早期退出设计大多通过向模型主干添加额外的网络分支来实现。然而，本研究表明，完全独立的小模型可以在对性能影响极小的情况下，替代大模型的相当一部分工作。将它们作为第一个退出点可以显著提高计算效率。通过搜索并采用最合适的小模型作为给定大模型的计算节省器，所提出的方法作为一种新颖且通用的模型压缩技术发挥作用。这一发现将帮助研究社区探索新的压缩方法，以应对快速发展的AI模型日益增长的计算需求。我们在ImageNet-1k分类任务上对该方法的评估表明，在各种现代视觉模型中，它能够将计算操作次数减少高达90%，而性能损失几乎可以忽略不计。本文的代码将在后续公开。