Utilizing massive web-scale datasets has led to unprecedented performance gains in machine learning models, but also imposes outlandish compute requirements for their training. In order to improve training and data efficiency, we here push the limits of pruning large-scale multimodal datasets for training CLIP-style models. Today's most effective pruning method on ImageNet clusters data samples into separate concepts according to their embedding and prunes away the most prototypical samples. We scale this approach to LAION and improve it by noting that the pruning rate should be concept-specific and adapted to the complexity of the concept. Using a simple and intuitive complexity measure, we are able to reduce the training cost to a quarter of regular training. By filtering from the LAION dataset, we find that training on a smaller set of high-quality data can lead to higher performance with significantly lower training costs. More specifically, we are able to outperform the LAION-trained OpenCLIP-ViT-B32 model on ImageNet zero-shot accuracy by 1.1p.p. while only using 27.7% of the data and training compute. Despite a strong reduction in training cost, we also see improvements on ImageNet dist. shifts, retrieval tasks and VTAB. On the DataComp Medium benchmark, we achieve a new state-of-the-art ImageNet zero-shot accuracy and a competitive average zero-shot accuracy on 38 evaluation tasks.

翻译：利用大规模网络级数据集虽然在机器学习模型中取得了前所未有的性能提升，但也带来了极其高昂的训练计算成本。为了提高训练效率和数据利用率，本文探索了剪枝大规模多模态数据集的关键极限，以训练CLIP类模型。当前基于ImageNet数据集最有效的剪枝方法，是根据数据样本的嵌入表示将其聚类为独立概念，并剔除最具原型特征的样本。我们将该方法扩展至LAION数据集，并通过引入概念特异性剪枝策略——即根据概念复杂度自适应调整剪枝率——对其进行了改进。借助简单直观的复杂度度量，我们成功将训练成本降低至常规训练的四分之一。通过对LAION数据集进行过滤，我们发现基于更小规模高质量数据集的训练，能以显著更低的计算成本获得更优性能。具体而言，在仅使用27.7%训练数据与计算资源的情况下，我们的模型在ImageNet零样本准确率上比基于LAION训练的OpenCLIP-ViT-B32模型提升了1.1个百分点。尽管训练成本大幅降低，模型在ImageNet分布偏移、检索任务及VTAB基准测试中同样呈现性能提升。在DataComp Medium基准测试中，我们不仅刷新了ImageNet零样本准确率的最高纪录，还在38项评估任务中取得了具有竞争力的平均零样本准确率。