Web-crawled datasets have enabled remarkable generalization capabilities in recent image-text models such as CLIP (Contrastive Language-Image pre-training) or Flamingo, but little is known about the dataset creation processes. In this work, we introduce a testbed of six publicly available data sources - YFCC, LAION, Conceptual Captions, WIT, RedCaps, Shutterstock - to investigate how pre-training distributions induce robustness in CLIP. We find that the performance of the pre-training data varies substantially across distribution shifts, with no single data source dominating. Moreover, we systematically study the interactions between these data sources and find that combining multiple sources does not necessarily yield better models, but rather dilutes the robustness of the best individual data source. We complement our empirical findings with theoretical insights from a simple setting, where combining the training data also results in diluted robustness. In addition, our theoretical model provides a candidate explanation for the success of the CLIP-based data filtering technique recently employed in the LAION dataset. Overall our results demonstrate that simply gathering a large amount of data from the web is not the most effective way to build a pre-training dataset for robust generalization, necessitating further study into dataset design. Code is available at https://github.com/mlfoundations/clip_quality_not_quantity.

翻译：网络爬取的数据集使得近期图像-文本模型（如CLIP（对比语言-图像预训练）或Flamingo）展现出卓越的泛化能力，但关于数据集创建过程的研究仍非常有限。本文引入一个包含六个公开数据源（YFCC、LAION、Conceptual Captions、WIT、RedCaps、Shutterstock）的测试平台，系统探究预训练数据分布如何影响CLIP的鲁棒性。研究发现，预训练数据的性能在不同分布偏移下存在显著差异，且没有任何单一数据源具有绝对优势。此外，我们系统研究了这些数据源之间的交互作用，发现组合多个数据源未必能提升模型性能，反而会削弱最优单一数据源的鲁棒性。我们通过一个简单场景的理论分析对实证结果进行补充，该场景中组合训练数据同样导致鲁棒性稀释。同时，理论模型为LAION数据集近期采用的基于CLIP的数据过滤技术的成功提供了合理解释。总体而言，我们的结果表明，仅从网络收集大量数据并非构建鲁棒泛化预训练数据集的最有效方法，亟需对数据集设计开展更深入的研究。相关代码已开源：https://github.com/mlfoundations/clip_quality_not_quantity。