Pretrained language models (LMs) are susceptible to generate text with nonfactual information. In this work, we measure and improve the factual accuracy of large-scale LMs for open-ended text generation. We design the FactualityPrompts test set and metrics to measure the factuality of LM generations. Based on that, we study the factual accuracy of LMs with parameter sizes ranging from 126M to 530B. Interestingly, we find that larger LMs are more factual than smaller ones, although a previous study suggests that larger LMs can be less truthful in terms of misconceptions. In addition, popular sampling algorithms (e.g., top-p) in open-ended text generation can harm the factuality due to the ''uniform randomness'' introduced at every sampling step. We propose the factual-nucleus sampling algorithm that dynamically adapts the randomness to improve the factuality of generation while maintaining quality. Furthermore, we analyze the inefficiencies of the standard training method in learning correct associations between entities from factual text corpus (e.g., Wikipedia). We propose a factuality-enhanced training method that uses TopicPrefix for better awareness of facts and sentence completion as the training objective, which can vastly reduce the factual errors. We release our code and FactualityPrompts benchmark at: https://github.com/nayeon7lee/FactualityPrompt.

翻译：预训练语言模型容易生成包含非事实信息的文本。本研究对大规模语言模型在开放式文本生成中的事实准确性进行度量和改进。我们设计了FactualityPrompts测试集和评估指标来衡量语言模型生成内容的事实性。在此基础上，我们研究了从126M到530B参数规模的模型的事实准确度。有趣的是，我们发现较大模型比较小模型更符合事实，尽管先前研究表明较大模型在误解方面可能更不真实。此外，开放式文本生成中常用的采样算法（如top-p）由于在每一步采样时引入"均匀随机性"会损害事实性。我们提出事实核采样算法，通过动态调整随机性来提升生成内容的事实性同时保持质量。进一步，我们分析了标准训练方法在学习事实文本语料（如维基百科）中实体间正确关联时的低效性。我们提出一种事实增强训练方法，采用TopicPrefix增强事实感知并以句子补全作为训练目标，能大幅减少事实错误。我们将代码及FactualityPrompts基准发布于：https://github.com/nayeon7lee/FactualityPrompt。