As increasingly sophisticated language models emerge, their trustworthiness becomes a pivotal issue, especially in tasks such as summarization and question-answering. Ensuring their responses are contextually grounded and faithful is challenging due to the linguistic diversity and the myriad of possible answers. In this paper, we introduce a novel approach to evaluate faithfulness of machine-generated text by computing the longest noncontinuous substring of the claim that is supported by the context, which we refer to as the Longest Supported Subsequence (LSS). Using a new human-annotated dataset, we finetune a model to generate LSS. We introduce a new method of evaluation and demonstrate that these metrics correlate better with human ratings when LSS is employed, as opposed to when it is not. Our proposed metric demonstrates an 18% enhancement over the prevailing state-of-the-art metric for faithfulness on our dataset. Our metric consistently outperforms other metrics on a summarization dataset across six different models. Finally, we compare several popular Large Language Models (LLMs) for faithfulness using this metric. We release the human-annotated dataset built for predicting LSS and our fine-tuned model for evaluating faithfulness.

翻译：随着日益复杂的语言模型的出现，其可信度成为关键问题，尤其在摘要生成和问答等任务中。由于语言多样性及可能答案的多样性，确保模型生成内容具有上下文依据和忠实性颇具挑战。本文提出一种新方法，通过计算声明中被上下文支持的最长非连续子序列（即最长支持子序列，LSS）来评估机器生成文本的忠实度。利用新构建的人工标注数据集，我们微调一个模型以生成LSS。我们引入一种新的评估方法，并证明使用LSS时，这些指标与人工评分之间的相关性优于未使用LSS时的情况。在我们数据集上，所提出的指标相较于主流的最先进忠实度指标实现了18%的提升。该指标在涵盖六种不同模型的摘要数据集上持续优于其他指标。最后，我们使用该指标对比了多种流行大型语言模型（LLM）的忠实度。我们公开了用于预测LSS的人工标注数据集及用于评估忠实度的微调模型。