Training learnable metrics using modern language models has recently emerged as a promising method for the automatic evaluation of machine translation. However, existing human evaluation datasets for text simplification have limited annotations that are based on unitary or outdated models, making them unsuitable for this approach. To address these issues, we introduce the SimpEval corpus that contains: SimpEval_past, comprising 12K human ratings on 2.4K simplifications of 24 past systems, and SimpEval_2022, a challenging simplification benchmark consisting of over 1K human ratings of 360 simplifications including GPT-3.5 generated text. Training on SimpEval, we present LENS, a Learnable Evaluation Metric for Text Simplification. Extensive empirical results show that LENS correlates much better with human judgment than existing metrics, paving the way for future progress in the evaluation of text simplification. We also introduce Rank and Rate, a human evaluation framework that rates simplifications from several models in a list-wise manner using an interactive interface, which ensures both consistency and accuracy in the evaluation process and is used to create the SimpEval datasets.

翻译：摘要：利用现代语言模型训练可学习指标，近期已成为机器翻译自动评估领域的一种有前景的方法。然而，现有文本简化任务的人类评估数据集基于单一或过时模型，标注数量有限，难以适用于此类方法。为解决这些问题，我们引入SimpEval语料库，包含：SimpEval_past（涵盖24个过往系统的2,400条简化文本的12,000条人工评分）以及SimpEval_2022（一个具有挑战性的简化基准，包含360条简化文本的超过1,000条人工评分，其中含GPT-3.5生成文本）。基于SimpEval训练，我们提出LENS——一种面向文本简化的可学习评估指标。广泛实验结果表明，LENS与人类判断的相关性显著优于现有指标，为文本简化评估领域的未来进展奠定基础。此外，我们提出Rank and Rate人类评估框架，通过交互式界面以列表方式对多个模型的简化结果进行评分，确保评估过程的一致性及准确性，并用于构建SimpEval数据集。