We introduce a method to measure uncertainty in large language models. For tasks like question answering, it is essential to know when we can trust the natural language outputs of foundation models. We show that measuring uncertainty in natural language is challenging because of "semantic equivalence" -- different sentences can mean the same thing. To overcome these challenges we introduce semantic entropy -- an entropy which incorporates linguistic invariances created by shared meanings. Our method is unsupervised, uses only a single model, and requires no modifications to off-the-shelf language models. In comprehensive ablation studies we show that the semantic entropy is more predictive of model accuracy on question answering data sets than comparable baselines.

翻译：我们提出一种度量大型语言模型不确定性的方法。对于问答等任务，了解何时可以信任基础模型的自然语言输出至关重要。我们证明了由于“语义等价性”（不同句子可能表达相同含义），在自然语言中度量不确定性具有挑战性。为克服这些挑战，我们引入了语义熵——一种融合了共享含义所产生语言不变性的熵。我们的方法是无监督的，仅使用单一模型，且无需对现有语言模型进行任何修改。在全面的消融研究中，我们展示了语义熵在预测问答数据集上的模型准确率方面优于可比基线方法。