Evaluating automatically-generated text summaries is a challenging task. While there have been many interesting approaches, they still fall short of human evaluations. We present RISE, a new approach for evaluating summaries by leveraging techniques from information retrieval. RISE is first trained as a retrieval task using a dual-encoder retrieval setup, and can then be subsequently utilized for evaluating a generated summary given an input document, without gold reference summaries. RISE is especially well suited when working on new datasets where one may not have reference summaries available for evaluation. We conduct comprehensive experiments on the SummEval benchmark (Fabbri et al., 2021) and the results show that RISE has higher correlation with human evaluations compared to many past approaches to summarization evaluation. Furthermore, RISE also demonstrates data-efficiency and generalizability across languages.

翻译：评估自动生成的文本摘要是具有挑战性的任务。尽管已有许多有趣的方案，但它们仍无法媲美人工评估。我们提出RISE，一种通过利用信息检索技术评估摘要的新方法。RISE首先采用双编码器检索框架作为检索任务进行训练，随后可用于评估给定输入文档所生成摘要的质量，且无需黄金参考摘要。该方法特别适用于缺乏参考摘要的新数据集评估场景。我们在SummEval基准测试（Fabbri et al., 2021）上开展全面实验，结果表明相比以往多种摘要评估方法，RISE与人工评估的相关性更高。此外，RISE还展现出数据高效性和跨语言泛化能力。