Large language models (LLMs) often generate content that contains factual errors when responding to fact-seeking prompts on open-ended topics. To benchmark a model's long-form factuality in open domains, we first use GPT-4 to generate LongFact, a prompt set comprising thousands of questions spanning 38 topics. We then propose that LLM agents can be used as automated evaluators for long-form factuality through a method which we call Search-Augmented Factuality Evaluator (SAFE). SAFE utilizes an LLM to break down a long-form response into a set of individual facts and to evaluate the accuracy of each fact using a multi-step reasoning process comprising sending search queries to Google Search and determining whether a fact is supported by the search results. Furthermore, we propose extending F1 score as an aggregated metric for long-form factuality. To do so, we balance the percentage of supported facts in a response (precision) with the percentage of provided facts relative to a hyperparameter representing a user's preferred response length (recall). Empirically, we demonstrate that LLM agents can achieve superhuman rating performance - on a set of ~16k individual facts, SAFE agrees with crowdsourced human annotators 72% of the time, and on a random subset of 100 disagreement cases, SAFE wins 76% of the time. At the same time, SAFE is more than 20 times cheaper than human annotators. We also benchmark thirteen language models on LongFact across four model families (Gemini, GPT, Claude, and PaLM-2), finding that larger language models generally achieve better long-form factuality. LongFact, SAFE, and all experimental code are available at https://github.com/google-deepmind/long-form-factuality.

翻译：大型语言模型（LLMs）在回答开放领域的事实性提示时，常常会生成包含事实错误的内容。为了在开放领域对模型的长篇事实性进行基准测试，我们首先利用GPT-4生成了LongFact——一个包含数千个问题、覆盖38个主题的提示集。进而我们提出，通过一种名为搜索增强事实评估器（SAFE）的方法，可以将LLM智能体用作长篇事实性的自动评估工具。SAFE利用LLM将长篇回复分解为一系列独立事实，并通过包含向Google搜索发送查询并判断事实是否被搜索结果支持的多步推理过程，评估每个事实的准确性。此外，我们提出将F1分数扩展为长篇事实性的聚合指标。为此，我们平衡了回复中受支持事实的百分比（精确率）与相对于代表用户偏好回复长度的超参数所提供的实际事实百分比（召回率）。实证表明，LLM智能体能够实现超越人类水平的评分性能——在大约16000个独立事实的集合上，SAFE与众包人工标注者的一致率达到72%，而在随机选取的100个分歧案例中，SAFE在76%的情况下胜出。同时，SAFE的成本比人工标注者低20倍以上。我们还对四个模型系列（Gemini、GPT、Claude和PaLM-2）中的十三种语言模型在LongFact上进行了基准测试，发现较大的语言模型通常能实现更好的长篇事实性。LongFact、SAFE及所有实验代码已公开于https://github.com/google-deepmind/long-form-factuality。