Progress in machine learning has been driven in large part by massive increases in data. However, large web-scale datasets such as LAION are largely uncurated beyond searches for exact duplicates, potentially leaving much redundancy. Here, we introduce SemDeDup, a method which leverages embeddings from pre-trained models to identify and remove semantic duplicates: data pairs which are semantically similar, but not exactly identical. Removing semantic duplicates preserves performance and speeds up learning. Analyzing a subset of LAION, we show that SemDeDup can remove 50% of the data with minimal performance loss, effectively halving training time. Moreover, performance increases out of distribution. Also, analyzing language models trained on C4, a partially curated dataset, we show that SemDeDup improves over prior approaches while providing efficiency gains. SemDeDup provides an example of how simple ways of leveraging quality embeddings can be used to make models learn faster with less data.

翻译：机器学习的进步在很大程度上得益于数据量的激增。然而，诸如LAION等大规模网络数据集除精确重复检测外基本未经整理，可能残留大量冗余数据。本文提出SemDeDup方法，该方法利用预训练模型的嵌入来识别并移除语义重复数据：即语义相似但不完全一致的数据对。移除语义重复数据可在保持性能的同时加速学习进程。通过对LAION子集的分析，我们证明SemDeDup在性能损失极小的情况下可移除50%的数据，使训练时间有效减半。此外，模型在分布外场景下的性能反而得到提升。同时，在基于部分整理数据集C4训练的语言模型分析中，SemDeDup在实现效率提升的同时优于现有方法。SemDeDup展示了如何通过简单利用优质嵌入，使模型以更少数据实现更快学习。