Large multimodal datasets have been instrumental in recent breakthroughs such as CLIP, Stable Diffusion, and GPT-4. At the same time, datasets rarely receive the same research attention as model architectures or training algorithms. To address this shortcoming in the machine learning ecosystem, we introduce DataComp, a benchmark where the training code is fixed and researchers innovate by proposing new training sets. We provide a testbed for dataset experiments centered around a new candidate pool of 12.8B image-text pairs from Common Crawl. Participants in our benchmark design new filtering techniques or curate new data sources and then evaluate their new dataset by running our standardized CLIP training code and testing on 38 downstream test sets. Our benchmark consists of multiple scales, with four candidate pool sizes and associated compute budgets ranging from 12.8M to 12.8B samples seen during training. This multi-scale design facilitates the study of scaling trends and makes the benchmark accessible to researchers with varying resources. Our baseline experiments show that the DataComp workflow is a promising way of improving multimodal datasets. We introduce DataComp-1B, a dataset created by applying a simple filtering algorithm to the 12.8B candidate pool. The resulting 1.4B subset enables training a CLIP ViT-L/14 from scratch to 79.2% zero-shot accuracy on ImageNet. Our new ViT-L/14 model outperforms a larger ViT-g/14 trained on LAION-2B by 0.7 percentage points while requiring 9x less training compute. We also outperform OpenAI's CLIP ViT-L/14 by 3.7 percentage points, which is trained with the same compute budget as our model. These gains highlight the potential for improving model performance by carefully curating training sets. We view DataComp-1B as only the first step and hope that DataComp paves the way toward the next generation of multimodal datasets.

翻译：大规模多模态数据集在近期突破性成果（如CLIP、Stable Diffusion和GPT-4）中发挥了关键作用。然而，与模型架构或训练算法相比，数据集很少受到同等研究关注。为弥补机器学习生态中的这一不足，我们引入DataComp基准——该基准固定训练代码，研究人员通过提出新训练集进行创新。我们围绕Common Crawl中128亿对图文候选池构建了一个数据集实验测试平台。基准参与者可设计新过滤技术或策划新数据源，并通过运行标准化CLIP训练代码、在38项下游测试集上评估其新数据集。该基准涵盖多尺度设计，包含四种候选池规模及相应计算预算（训练样本量从1280万至128亿不等）。多尺度设计便于研究缩放趋势，并让不同资源条件的研究者均能参与。基线实验表明，DataComp工作流是改进多模态数据集的有效途径。我们推出DataComp-1B数据集——通过对128亿候选池应用简单过滤算法生成。该14亿子集支持从零训练CLIP ViT-L/14模型，在ImageNet上达到79.2%的零样本准确率。新ViT-L/14模型在性能上超越基于LAION-2B训练的更大规模ViT-g/14模型0.7个百分点，同时训练计算量减少9倍。与采用相同计算预算的OpenAI CLIP ViT-L/14相比，我们提升3.7个百分点。这些成果凸显了通过精心策划训练集来提升模型性能的潜力。我们将DataComp-1B视为第一步，期待DataComp为下一代多模态数据集铺平道路。