Concerns about reproducibility in artificial intelligence (AI) have emerged, as researchers have reported unsuccessful attempts to directly reproduce published findings in the field. Replicability, the ability to affirm a finding using the same procedures on new data, has not been well studied. In this paper, we examine both reproducibility and replicability of a corpus of 16 papers on table structure recognition (TSR), an AI task aimed at identifying cell locations of tables in digital documents. We attempt to reproduce published results using codes and datasets provided by the original authors. We then examine replicability using a dataset similar to the original as well as a new dataset, GenTSR, consisting of 386 annotated tables extracted from scientific papers. Out of 16 papers studied, we reproduce results consistent with the original in only four. Two of the four papers are identified as replicable using the similar dataset under certain IoU values. No paper is identified as replicable using the new dataset. We offer observations on the causes of irreproducibility and irreplicability. All code and data are available on Codeocean at https://codeocean.com/capsule/6680116/tree.

翻译：人工智能（AI）领域对可重复性的担忧日益凸显，研究人员报告了在该领域直接复现已发表研究成果时屡遭失败的案例。复现性——即通过相同流程在新数据上验证研究结论的能力——尚未得到充分研究。本文针对表格结构识别（TSR）这一旨在定位数字文档中表格单元格位置的AI任务，系统考察了16篇相关论文的可重复性与复现性。我们首先使用原作者提供的代码与数据集尝试复现已发表结果，继而采用与原数据集类似的相似数据集以及从科学论文中提取的包含386个标注表格的新数据集GenTSR进行复现性检验。研究结果表明：在16篇论文中，仅4篇可复现出与原文一致的结果；其中2篇在特定交并比阈值条件下可通过相似数据集实现复现，但无任何论文能通过新数据集实现复现。我们针对不可重复性与不可复现性的成因提出了相关见解。所有代码与数据均托管于Codeocean平台：https://codeocean.com/capsule/6680116/tree。