Constructing a timeline requires identifying the chronological order of events in an article. In prior timeline construction datasets, temporal orders are typically annotated by either event-to-time anchoring or event-to-event pairwise ordering, both of which suffer from missing temporal information. To mitigate the issue, we develop a new evaluation dataset, TimeSET, consisting of single-document timelines with document-level order annotation. TimeSET features saliency-based event selection and partial ordering, which enable a practical annotation workload. Aiming to build better automatic timeline construction systems, we propose a novel evaluation framework to compare multiple task formulations with TimeSET by prompting open LLMs, i.e., Llama 2 and Flan-T5. Considering that identifying temporal orders of events is a core subtask in timeline construction, we further benchmark open LLMs on existing event temporal ordering datasets to gain a robust understanding of their capabilities. Our experiments show that (1) NLI formulation with Flan-T5 demonstrates a strong performance among others, while (2) timeline construction and event temporal ordering are still challenging tasks for few-shot LLMs. Our code and data are available at https://github.com/kimihiroh/timeset.

翻译：构建时间线需要识别文章中事件的时序关系。既有时间线构建数据集中，通常通过事件-时间锚定或事件间两两排序两种方式标注时序关系，这两种方法均存在时序信息缺失的问题。为缓解该问题，我们开发了新型评估数据集TimeSET，该数据集包含带有文档级排序标注的单文档时间线。TimeSET采用基于显著度的事件选择与偏序标注策略，使实际标注工作量保持可操作性。为构建更优的自动时间线构建系统，我们提出新型评估框架，通过提示开放型LLM（即Llama 2与Flan-T5）比较TimeSET上的多种任务公式化方案。考虑到事件时序识别是时间线构建的核心子任务，我们进一步在现有事件时序排序数据集上对开放型LLM进行基准测试，以全面把握其能力。实验表明：（1）基于Flan-T5的自然语言推理公式化方案在多方案中表现优异；（2）对于少样本LLM而言，时间线构建与事件时序排序仍是具有挑战性的任务。我们的代码与数据已开源至https://github.com/kimihiroh/timeset。