Contrastive Language-Image Pre-training (CLIP) is an approach that has advanced research and applications in computer vision, fueling modern recognition systems and generative models. We believe that the main ingredient to the success of CLIP is its data and not the model architecture or pre-training objective. However, CLIP only provides very limited information about its data and how it has been collected, leading to works that aim to reproduce CLIP's data by filtering with its model parameters. In this work, we intend to reveal CLIP's data curation approach and in our pursuit of making it open to the community introduce Metadata-Curated Language-Image Pre-training (MetaCLIP). MetaCLIP takes a raw data pool and metadata (derived from CLIP's concepts) and yields a balanced subset over the metadata distribution. Our experimental study rigorously isolates the model and training settings, concentrating solely on data. MetaCLIP applied to CommonCrawl with 400M image-text data pairs outperforms CLIP's data on multiple standard benchmarks. In zero-shot ImageNet classification, MetaCLIP achieves 70.8% accuracy, surpassing CLIP's 68.3% on ViT-B models. Scaling to 1B data, while maintaining the same training budget, attains 72.4%. Our observations hold across various model sizes, exemplified by ViT-H achieving 80.5%, without any bells-and-whistles. Curation code and training data distribution on metadata is made available at https://github.com/facebookresearch/MetaCLIP.

翻译：对比语言-图像预训练（CLIP）是一种推动计算机视觉研究与应用的范式，支撑了现代识别系统与生成模型。我们认为CLIP成功的关键在于其数据，而非模型架构或预训练目标。然而，CLIP仅提供了关于其数据及其收集方式的极为有限的信息，这促使许多研究尝试通过其模型参数筛选数据，以复现CLIP的数据集。本研究旨在揭示CLIP的数据策展方法，并在此过程中向社区开源元数据策展语言-图像预训练（MetaCLIP）。MetaCLIP利用原始数据池和元数据（源自CLIP的概念）生成一个基于元数据分布均衡的子集。我们的实验严格隔离了模型与训练设置，仅聚焦于数据本身。将MetaCLIP应用于包含4亿图文对的CommonCrawl数据集后，其在多项标准基准测试中均优于CLIP数据。在零样本ImageNet分类任务中，MetaCLIP在ViT-B模型上达到70.8%的准确率，超越CLIP的68.3%。在保持相同训练预算的前提下，将数据扩展至10亿规模后，准确率提升至72.4%。这一结论在不同模型规模下均成立，例如ViT-H在无需任何额外技巧的情况下实现80.5%的准确率。策展代码及基于元数据的训练数据分布已发布于https://github.com/facebookresearch/MetaCLIP。