Benchmark datasets for table structure recognition (TSR) must be carefully processed to ensure they are annotated consistently. However, even if a dataset's annotations are self-consistent, there may be significant inconsistency across datasets, which can harm the performance of models trained and evaluated on them. In this work, we show that aligning these benchmarks$\unicode{x2014}$removing both errors and inconsistency between them$\unicode{x2014}$improves model performance significantly. We demonstrate this through a data-centric approach where we adopt a single model architecture, the Table Transformer (TATR), that we hold fixed throughout. Baseline exact match accuracy for TATR evaluated on the ICDAR-2013 benchmark is 65% when trained on PubTables-1M, 42% when trained on FinTabNet, and 69% combined. After reducing annotation mistakes and inter-dataset inconsistency, performance of TATR evaluated on ICDAR-2013 increases substantially to 75% when trained on PubTables-1M, 65% when trained on FinTabNet, and 81% combined. We show through ablations over the modification steps that canonicalization of the table annotations has a significantly positive effect on performance, while other choices balance necessary trade-offs that arise when deciding a benchmark dataset's final composition. Overall we believe our work has significant implications for benchmark design for TSR and potentially other tasks as well. All dataset processing and training code will be released.

翻译：表格结构识别（TSR）基准数据集必须经过精心处理，以确保其标注的一致性。然而，即使单个数据集的标注内部一致，不同数据集之间仍可能存在显著不一致性，这会对基于这些数据集训练和评估的模型性能产生负面影响。本研究证明，对齐这些基准数据集——消除其中的错误及数据集间的不一致性——能显著提升模型性能。我们采用数据为中心的方法进行论证，始终使用固定不变的单一模型架构——表格Transformer（TATR）。基线精确匹配准确率显示：在ICDAR-2013基准上评估时，基于PubTables-1M训练的TATR为65%，基于FinTabNet训练的为42%，两者联合训练为69%。在减少标注错误和跨数据集不一致性后，基于ICDAR-2013评估的TATR性能大幅提升：基于PubTables-1M训练时达75%，基于FinTabNet训练时达65%，联合训练时达81%。通过逐项消融实验，我们证明表格标注的规范化对性能有显著正向影响，而其他步骤则需平衡决定基准数据集最终组成时出现的必要权衡。总体而言，我们认为本研究对TSR乃至其他任务的基准设计具有重要启示。所有数据集处理代码和训练代码将予以公开。