Reliable uncertainty quantification (UQ) is essential for deploying large language models (LLMs) in safety-critical scenarios, as it enables them to abstain from responding when uncertain, thereby avoiding hallucinations, i.e., plausible yet factually incorrect responses. However, while semantic UQ methods have achieved advanced performance, they overlook latent semantic structural information that could enable more precise uncertainty estimates. In this paper, we propose \underline{Se}mantic \underline{S}tructural \underline{E}ntropy ({SeSE}), a principled black-box UQ framework applicable to both open- and closed-source LLMs. To reveal the intrinsic structure of the semantic space, SeSE constructs its optimal hierarchical abstraction through an encoding tree with minimal structural entropy. The structural entropy of this encoding tree thus quantifies the inherent uncertainty within LLM semantic space after optimal compression. Additionally, unlike existing methods that primarily focus on simple short-form generation, we extent SeSE to provide interpretable, granular uncertainty estimation for long-form outputs. We theoretically prove that SeSE generalizes semantic entropy, the gold standard for UQ in LLMs, and empirically demonstrate its superior performance over strong baselines across 24 model-dataset combinations.

翻译：可靠的不确定性量化对于在安全关键场景中部署大型语言模型至关重要，因为它使模型能够在不确定时主动弃答，从而避免产生幻觉——即看似合理但事实错误的回答。然而，尽管语义不确定性量化方法已取得先进性能，它们忽略了潜在的语义结构信息，而这些信息可能实现更精确的不确定性估计。本文提出 \underline{Se}mantic \underline{S}tructural \underline{E}ntropy（{SeSE}），一个适用于开源和闭源大型语言模型的原则性黑盒不确定性量化框架。为揭示语义空间的内在结构，SeSE通过构建具有最小结构熵的编码树，形成语义空间的最优层次抽象。该编码树的结构熵从而量化了最优压缩后大型语言模型语义空间的内在不确定性。此外，与现有主要关注简单短文本生成的方法不同，我们将SeSE扩展至为长文本输出提供可解释的、细粒度的不确定性估计。我们从理论上证明SeSE推广了语义熵——大型语言模型不确定性量化的黄金标准，并通过在24种模型-数据集组合上的实验，实证展示了其相对于强基线的优越性能。