Large Language Models (LLMs) have demonstrated remarkable proficiency in various natural language generation (NLG) tasks. Previous studies suggest that LLMs' generation process involves uncertainty. However, existing approaches to uncertainty estimation mainly focus on sequence-level uncertainty, overlooking individual pieces of information within sequences. These methods fall short in separately assessing the uncertainty of each component in a sequence. In response, we propose a novel framework for Concept-Level Uncertainty Estimation (CLUE) for LLMs. We leverage LLMs to convert output sequences into concept-level representations, breaking down sequences into individual concepts and measuring the uncertainty of each concept separately. We conduct experiments to demonstrate that CLUE can provide more interpretable uncertainty estimation results compared with sentence-level uncertainty, and could be a useful tool for various tasks such as hallucination detection and story generation.

翻译：大语言模型（LLMs）在各种自然语言生成（NLG）任务中展现出卓越的能力。先前研究表明，LLMs的生成过程涉及不确定性。然而，现有的不确定性估计方法主要关注序列级不确定性，忽略了序列内部的独立信息单元。这些方法无法对序列中每个组成部分的不确定性进行分别评估。为此，我们提出了一种面向LLMs的概念级不确定性估计（CLUE）新框架。我们利用LLMs将输出序列转化为概念级表示，将序列分解为独立概念并分别测量每个概念的不确定性。实验表明，与句子级不确定性相比，CLUE能够提供更具可解释性的不确定性估计结果，并可能成为幻觉检测和故事生成等多种任务的有效工具。