Progress in machine learning has been driven in large part by massive increases in data. However, large web-scale datasets such as LAION are largely uncurated beyond searches for exact duplicates, potentially leaving much redundancy. Here, we introduce SemDeDup, a method which leverages embeddings from pre-trained models to identify and remove semantic duplicates: data pairs which are semantically similar, but not exactly identical. Removing semantic duplicates preserves performance and speeds up learning. Analyzing a subset of LAION, we show that SemDeDup can remove 50% of the data with minimal performance loss, effectively halving training time. Moreover, performance increases out of distribution. Also, analyzing language models trained on C4, a partially curated dataset, we show that SemDeDup improves over prior approaches while providing efficiency gains. SemDeDup provides an example of how simple ways of leveraging quality embeddings can be used to make models learn faster with less data.

翻译：机器学习的发展在很大程度上得益于数据的巨量增长。然而，诸如LAION之类的大规模网络数据集，除了进行精确重复项搜索外，大多未经过精心整理，可能遗留大量冗余信息。本文提出SemDeDup方法，利用预训练模型生成的嵌入向量来识别并移除语义重复项——即语义相似但非完全一致的数据对。移除语义重复项既能保持模型性能，又能加速学习过程。通过对LAION子集的分析，我们证明SemDeDup可在性能损失极小的情况下移除50%的数据，有效将训练时间减半。此外，在分布外场景下模型性能反而有所提升。同时，针对经过部分整理的C4数据集训练的語言模型，我们证明SemDeDup在提升效率的同时，优于此前方法。SemDeDup展示了如何通过简单利用高质量嵌入向量，使模型用更少数据实现更快学习。