Maintaining factual consistency is a critical issue in abstractive text summarisation, however, it cannot be assessed by traditional automatic metrics used for evaluating text summarisation, such as ROUGE scoring. Recent efforts have been devoted to developing improved metrics for measuring factual consistency using pre-trained language models, but these metrics have restrictive token limits, and are therefore not suitable for evaluating long document text summarisation. Moreover, there is limited research and resources available for evaluating whether existing automatic evaluation metrics are fit for purpose when applied in long document settings. In this work, we evaluate the efficacy of automatic metrics for assessing the factual consistency of long document text summarisation. We create a human-annotated data set for evaluating automatic factuality metrics, LongSciVerify, which contains fine-grained factual consistency annotations for long document summaries from the scientific domain. We also propose a new evaluation framework, LongDocFACTScore, which is suitable for evaluating long document summarisation. This framework allows metrics to be efficiently extended to any length document and outperforms existing state-of-the-art metrics in its ability to correlate with human measures of factuality when used to evaluate long document summarisation data sets. We make our code and LongSciVerify data set publicly available: https://github.com/jbshp/LongDocFACTScore.

翻译：在抽象文本摘要中保持事实一致性是一个关键问题，然而传统的自动摘要评估指标（如ROUGE评分）无法对此进行评估。近期研究致力于利用预训练语言模型开发改进的度量标准以衡量事实一致性，但这些指标存在严格的令牌长度限制，因此不适用于长文档文本摘要的评估。此外，针对现有自动评估指标在长文档场景下的适用性，相关研究和资源仍较为有限。本研究评估了自动指标在长文档文本摘要事实一致性评估中的有效性。我们创建了用于评估自动事实性指标的人工标注数据集LongSciVerify，该数据集包含来自科学领域的长文档摘要的细粒度事实一致性标注。同时，我们提出了适用于长文档摘要评估的新框架LongDocFACTScore。该框架允许评估指标高效扩展至任意长度文档，在评估长文档摘要数据集时，其与人工事实性评估的相关性优于现有最先进指标。我们的代码及LongSciVerify数据集已公开：https://github.com/jbshp/LongDocFACTScore。