Entity resolution (ER) remains a significant challenge in data management, especially when dealing with large datasets. This paper introduces MERAI (Massive Entity Resolution using AI), a robust and efficient pipeline designed to address record deduplication and linkage issues in high-volume datasets at an enterprise level. The pipeline's resilience and accuracy have been validated through various large-scale record deduplication and linkage projects. To evaluate MERAI's performance, we compared it with two well-known entity resolution libraries, Dedupe and Splink. While Dedupe failed to scale beyond 2 million records due to memory constraints, MERAI successfully processed datasets of up to 15.7 million records and produced accurate results across all experiments. Experimental data demonstrates that MERAI outperforms both baseline systems in terms of matching accuracy, with consistently higher F1 scores in both deduplication and record linkage tasks. MERAI offers a scalable and reliable solution for enterprise-level large-scale entity resolution, ensuring data integrity and consistency in real-world applications.

翻译：实体解析（ER）在数据管理领域仍是一项重大挑战，尤其是在处理大规模数据集时。本文介绍了MERAI（基于人工智能的大规模实体解析），这是一个专为解决企业级高容量数据集中的记录去重与关联问题而设计的鲁棒高效流水线。该流水线的鲁棒性和准确性已通过多个大规模记录去重与关联项目得到验证。为评估MERAI的性能，我们将其与两个知名的实体解析库——Dedupe和Splink进行了比较。Dedupe由于内存限制无法扩展至200万条记录以上，而MERAI成功处理了高达1570万条记录的数据集，并在所有实验中均取得了准确结果。实验数据表明，MERAI在匹配准确率方面优于两个基线系统，在去重和记录关联任务中均持续获得更高的F1分数。MERAI为企业级大规模实体解析提供了一个可扩展且可靠的解决方案，确保了实际应用中的数据完整性与一致性。