Contrastive pretraining of image-text foundation models, such as CLIP, demonstrated excellent zero-shot performance and improved robustness on a wide range of downstream tasks. However, these models utilize large transformer-based encoders with significant memory and latency overhead which pose challenges for deployment on mobile devices. In this work, we introduce MobileCLIP -- a new family of efficient image-text models optimized for runtime performance along with a novel and efficient training approach, namely multi-modal reinforced training. The proposed training approach leverages knowledge transfer from an image captioning model and an ensemble of strong CLIP encoders to improve the accuracy of efficient models. Our approach avoids train-time compute overhead by storing the additional knowledge in a reinforced dataset. MobileCLIP sets a new state-of-the-art latency-accuracy tradeoff for zero-shot classification and retrieval tasks on several datasets. Our MobileCLIP-S2 variant is 2.3$\times$ faster while more accurate compared to previous best CLIP model based on ViT-B/16. We further demonstrate the effectiveness of our multi-modal reinforced training by training a CLIP model based on ViT-B/16 image backbone and achieving +2.9% average performance improvement on 38 evaluation benchmarks compared to the previous best. Moreover, we show that the proposed approach achieves 10$\times$-1000$\times$ improved learning efficiency when compared with non-reinforced CLIP training.

翻译：基于对比学习的图像-文本基础模型（如CLIP）在零样本任务和下游任务鲁棒性方面展现了卓越性能。然而，这类模型采用基于Transformer的大型编码器，其显著的内存占用和延迟开销给移动设备部署带来了挑战。本文提出MobileCLIP——一类针对运行时性能优化的高效图文模型新家族，并配套提出一种新颖高效的训练方法——多模态强化训练。该方法通过从图像描述模型和强CLIP编码器集成中迁移知识，提升高效模型的准确率。我们通过将额外知识存储在强化数据集中，避免了训练阶段的计算开销。MobileCLIP在多个数据集上的零样本分类与检索任务中，建立了延迟-精度权衡的新标杆。我们的MobileCLIP-S2变体相较基于ViT-B/16的此前最佳CLIP模型，速度提升2.3倍且精度更高。进一步，我们通过训练基于ViT-B/16图像骨干的CLIP模型验证了多模态强化训练的有效性，在38项评测基准上平均性能较此前最佳提升+2.9%。此外，与未采用强化训练的CLIP方法相比，所提出的方案在38项评测基准上实现了10倍至1000倍的学习效率提升。