Existing metrics for evaluating the factuality of long-form text, such as FACTSCORE (Min et al., 2023) and SAFE (Wei et al., 2024), decompose an input text into "atomic claims" and verify each against a knowledge base like Wikipedia. These metrics are not suitable for most generation tasks because they assume that every claim is verifiable (i.e., can plausibly be proven true or false). We address this issue with VERISCORE, a metric for diverse long-form generation tasks that contain both verifiable and unverifiable content. VERISCORE can be effectively implemented with either closed or fine-tuned open-weight language models, and human evaluation confirms that VERISCORE's extracted claims are more sensible than those from competing methods across eight different long-form tasks. We use VERISCORE to evaluate generations from 16 different models across multiple long-form tasks and find that while GPT-4o is the best-performing model overall, open-weight models such as Mixtral-8x22 are closing the gap. We show that an LM's VERISCORE on one task (e.g., biography generation) does not necessarily correlate to its VERISCORE on a different task (e.g., long-form QA), highlighting the need for expanding factuality evaluation across tasks with varying fact density.

翻译：现有评估长文本事实性的指标，如FACTSCORE（Min等人，2023）和SAFE（Wei等人，2024），将输入文本分解为“原子声明”并依据维基百科等知识库逐一验证。这些指标并不适用于大多数生成任务，因为它们假设每个声明都是可验证的（即可能被证实为真或假）。我们通过VERISCORE解决这一问题，该指标适用于包含可验证与不可验证内容的多样化长文本生成任务。VERISCORE可通过闭源模型或微调的开源权重语言模型有效实现，人工评估证实，在八种不同的长文本任务中，VERISCORE提取的声明比竞争方法更合理。我们使用VERISCORE评估了16个不同模型在多个长文本任务中的生成结果，发现虽然GPT-4o是整体表现最佳的模型，但Mixtral-8x22等开源权重模型正在缩小差距。研究表明，语言模型在某一任务（如传记生成）上的VERISCORE得分与其在另一任务（如长文本问答）上的得分未必相关，这凸显了需要将事实性评估扩展至具有不同事实密度的多样化任务中。