Contrastive Language-Image Pre-training (CLIP) is an approach that has advanced research and applications in computer vision, fueling modern recognition systems and generative models. We believe that the main ingredient to the success of CLIP is its data and not the model architecture or pre-training objective. However, CLIP only provides very limited information about its data and how it has been collected, leading to works that aim to reproduce CLIP's data by filtering with its model parameters. In this work, we intend to reveal CLIP's data curation approach and in our pursuit of making it open to the community introduce Metadata-Curated Language-Image Pre-training (MetaCLIP). MetaCLIP takes a raw data pool and metadata (derived from CLIP's concepts) and yields a balanced subset over the metadata distribution. Our experimental study rigorously isolates the model and training settings, concentrating solely on data. MetaCLIP applied to CommonCrawl with 400M image-text data pairs outperforms CLIP's data on multiple standard benchmarks. In zero-shot ImageNet classification, MetaCLIP achieves 70.8% accuracy, surpassing CLIP's 68.3% on ViT-B models. Scaling to 1B data, while maintaining the same training budget, attains 72.4%. Our observations hold across various model sizes, exemplified by ViT-H achieving 80.5%, without any bells-and-whistles. Curation code and training data distribution on metadata is made available at https://github.com/facebookresearch/MetaCLIP.

翻译：对比语言-图像预训练（CLIP）是一种推动计算机视觉研究与应用进步的方法，为现代识别系统和生成模型提供了动力。我们认为CLIP成功的关键在于其数据，而非模型架构或预训练目标。然而，CLIP仅提供了关于其数据及收集方式的极其有限的信息，这导致了一些工作试图通过使用其模型参数来过滤数据以复现CLIP的数据集。在本研究中，我们旨在揭示CLIP的数据整理方法，并在致力于向社区开放的过程中，引入元数据驱动的语言-图像预训练（MetaCLIP）。MetaCLIP接收原始数据池和元数据（源自CLIP的概念），并在元数据分布上生成一个平衡的子集。我们的实验研究严格隔离了模型和训练设置，仅聚焦于数据本身。将MetaCLIP应用于包含4亿图像-文本对的CommonCrawl数据集，其在多个标准基准测试中表现优于CLIP的数据。在零样本ImageNet分类中，MetaCLIP在ViT-B模型上达到70.8%的准确率，超过了CLIP的68.3%。在保持相同训练预算的情况下，将数据规模扩展至10亿，准确率达到72.4%。我们的观察结果适用于各种模型规模，例如ViT-H在无需任何额外技巧的情况下实现了80.5%的准确率。数据整理代码及元数据上的训练数据分布已在https://github.com/facebookresearch/MetaCLIP 开源。