Entity resolution (ER) is the process of identifying records that refer to the same entities within one or across multiple databases. Numerous techniques have been developed to tackle ER challenges over the years, with recent emphasis placed on machine and deep learning methods for the matching phase. However, the quality of the benchmark datasets typically used in the experimental evaluations of learning-based matching algorithms has not been examined in the literature. To cover this gap, we propose four different approaches to assessing the difficulty and appropriateness of 13 established datasets: two theoretical approaches, which involve new measures of linearity and existing measures of complexity, and two practical approaches: the difference between the best non-linear and linear matchers, as well as the difference between the best learning-based matcher and the perfect oracle. Our analysis demonstrates that most of the popular datasets pose rather easy classification tasks. As a result, they are not suitable for properly evaluating learning-based matching algorithms. To address this issue, we propose a new methodology for yielding benchmark datasets. We put it into practice by creating four new matching tasks, and we verify that these new benchmarks are more challenging and therefore more suitable for further advancements in the field.

翻译：实体解析（Entity Resolution, ER）是识别同一数据库或跨多个数据库中指向同一实体的记录的过程。多年来，研究者开发了众多技术以应对ER挑战，近期重点在于将机器学习与深度学习方法应用于匹配阶段。然而，文献中尚未对学习型匹配算法实验评估中常用的基准数据集质量进行检验。为填补此空白，我们提出了四种评估13个已有数据集难度与适宜性的方法：两种理论方法（涉及新的线性度量与现有复杂度度量）及两种实践方法（最优非线性匹配器与线性匹配器的差异，以及最优学习型匹配器与完美预言机之间的差异）。分析表明，大多数流行数据集构成了相当简单的分类任务，因此不适用于恰当评估学习型匹配算法。为解决此问题，我们提出了一种生成基准数据集的新方法。通过创建四个新匹配任务并将其付诸实践，我们验证了这些新基准更具挑战性，因而更适宜推动该领域的进一步发展。