In this paper, we propose a novel cross-modal distillation method, called TinyCLIP, for large-scale language-image pre-trained models. The method introduces two core techniques: affinity mimicking and weight inheritance. Affinity mimicking explores the interaction between modalities during distillation, enabling student models to mimic teachers' behavior of learning cross-modal feature alignment in a visual-linguistic affinity space. Weight inheritance transmits the pre-trained weights from the teacher models to their student counterparts to improve distillation efficiency. Moreover, we extend the method into a multi-stage progressive distillation to mitigate the loss of informative weights during extreme compression. Comprehensive experiments demonstrate the efficacy of TinyCLIP, showing that it can reduce the size of the pre-trained CLIP ViT-B/32 by 50%, while maintaining comparable zero-shot performance. While aiming for comparable performance, distillation with weight inheritance can speed up the training by 1.4 - 7.8 $\times$ compared to training from scratch. Moreover, our TinyCLIP ViT-8M/16, trained on YFCC-15M, achieves an impressive zero-shot top-1 accuracy of 41.1% on ImageNet, surpassing the original CLIP ViT-B/16 by 3.5% while utilizing only 8.9% parameters. Finally, we demonstrate the good transferability of TinyCLIP in various downstream tasks. Code and models will be open-sourced at https://aka.ms/tinyclip.

翻译：本文提出了一种名为TinyCLIP的新型跨模态蒸馏方法，适用于大规模语言-图像预训练模型。该方法引入两项核心技术：亲和模仿与权重继承。亲和模仿在蒸馏过程中探索模态间的交互，使学生模型能够在视觉-语言亲和空间中模仿教师学习跨模态特征对齐的行为。权重继承则将教师模型的预训练权重传递给学生模型，以提高蒸馏效率。此外，我们将该方法扩展为多阶段渐进式蒸馏，以缓解极端压缩过程中信息权重的损失。综合实验证明了TinyCLIP的有效性：它可将预训练CLIP ViT-B/32的规模缩减50%，同时保持相当的零样本性能。在追求可比性能时，采用权重继承的蒸馏相比从头训练可加速1.4-7.8倍。此外，我们在YFCC-15M上训练的TinyCLIP ViT-8M/16在ImageNet上实现了令人瞩目的41.1%零样本Top-1准确率，以仅8.9%的参数量超越原始CLIP ViT-B/16达3.5%。最后，我们展示了TinyCLIP在多种下游任务中的良好迁移能力。代码与模型将开源至https://aka.ms/tinyclip。