Temporal relation extraction models have thus far been hindered by a number of issues in existing temporal relation-annotated news datasets, including: (1) low inter-annotator agreement due to the lack of specificity of their annotation guidelines in terms of what counts as a temporal relation; (2) the exclusion of long-distance relations within a given document (those spanning across different paragraphs); and (3) the exclusion of events that are not centred on verbs. This paper aims to alleviate these issues by presenting a new annotation scheme that clearly defines the criteria based on which temporal relations should be annotated. Additionally, the scheme includes events even if they are not expressed as verbs (e.g., nominalised events). Furthermore, we propose a method for annotating all temporal relations -- including long-distance ones -- which automates the process, hence reducing time and manual effort on the part of annotators. The result is a new dataset, the TIMELINE corpus, in which improved inter-annotator agreement was obtained, in comparison with previously reported temporal relation datasets. We report the results of training and evaluating baseline temporal relation extraction models on the new corpus, and compare them with results obtained on the widely used MATRES corpus.

翻译：时间关系抽取模型一直受到现有时间关系标注新闻数据集中若干问题的制约，包括：(1) 标注指南对时间关系界定标准缺乏明确性导致标注者间一致性较低；(2) 忽略同一文档内跨段落的长距离关系；(3) 未涵盖不以动词为中心的事件。本文通过提出一套新的标注方案来缓解上述问题，该方案明确定义了时间关系标注的判定标准。此外，该方案将非动词形式表述的事件（如名词化事件）也纳入标注范围。我们进一步提出一种全自动标注所有时间关系（包括长距离关系）的方法，从而减少标注者的时间投入和人工负担。由此生成的新数据集TIMELINE语料库相较于已有时间关系数据集，获得了更高的标注者间一致性。我们报告了在该新语料库上训练并评估基线时间关系抽取模型的结果，并与广泛使用的MATRES语料库上的实验结果进行了对比。