In the era of big data, the issue of data quality has become increasingly prominent. One of the main challenges is the problem of duplicate data, which can arise from repeated entry or the merging of multiple data sources. These "dirty data" problems can significantly limit the effective application of big data. To address the issue of data deduplication, we propose a pre-trained deduplication model based on active learning, which is the first work that utilizes active learning to address the problem of deduplication at the semantic level. The model is built on a pre-trained Transformer and fine-tuned to solve the deduplication problem as a sequence to classification task, which firstly integrate the transformer with active learning into an end-to-end architecture to select the most valuable data for deduplication model training, and also firstly employ the R-Drop method to perform data augmentation on each round of labeled data, which can reduce the cost of manual labeling and improve the model's performance. Experimental results demonstrate that our proposed model outperforms previous state-of-the-art (SOTA) for deduplicated data identification, achieving up to a 28% improvement in Recall score on benchmark datasets.

翻译：在大数据时代，数据质量问题日益凸显，其中重复数据是主要挑战之一，可能源于重复录入或多数据源合并。这些“脏数据”问题会严重限制大数据的有效应用。为解决数据去重问题，我们提出了一种基于主动学习的预训练去重模型，这是首个利用主动学习在语义层面解决去重问题的研究工作。该模型基于预训练的Transformer构建，并通过微调将去重问题转化为序列到分类的任务，首次将Transformer与主动学习集成到端到端架构中，以选取最有价值的数据进行去重模型训练，同时首次采用R-Drop方法对每轮标注数据进行数据增强，从而降低人工标注成本并提升模型性能。实验结果表明，我们提出的模型在重复数据识别任务上优于先前的最优方法，在基准数据集上的召回率最高提升了28%。