Uncertainty quantification (UQ) is a perspective approach to detecting Large Language Model (LLM) hallucinations and low quality output. In this work, we address one of the challenges of UQ in generation tasks that arises from the conditional dependency between the generation steps of an LLM. We propose to learn this dependency from data. We train a regression model, which target variable is the gap between the conditional and the unconditional generation confidence. During LLM inference, we use this learned conditional dependency model to modulate the uncertainty of the current generation step based on the uncertainty of the previous step. Our experimental evaluation on nine datasets and three LLMs shows that the proposed method is highly effective for uncertainty quantification, achieving substantial improvements over rivaling approaches.

翻译：不确定性量化（UQ）是检测大型语言模型（LLM）幻觉及低质量输出的一种前瞻性方法。在本工作中，我们解决了生成任务中UQ的一个挑战，该挑战源于LLM生成步骤之间的条件依赖性。我们提出从数据中学习这种依赖关系。我们训练了一个回归模型，其目标变量是条件生成置信度与无条件生成置信度之间的差距。在LLM推理过程中，我们利用这个学习到的条件依赖模型，基于前一步骤的不确定性来调整当前生成步骤的不确定性。我们在九个数据集和三个LLM上进行的实验评估表明，所提出的方法对于不确定性量化非常有效，相较于竞争方法取得了显著改进。