Many black-box techniques for quantifying the uncertainty of large language models (LLMs) rely on repeated LLM sampling, which can be computationally expensive. Therefore, practical applicability demands reliable estimation from few samples. Semantic entropy (SE) is a popular sample-based uncertainty estimator with a discrete formulation attractive for the black-box setting. Recent extensions of SE exhibit improved LLM hallucination detection, but do so with less interpretable methods that admit additional hyperparameters. For this reason, we revisit the canonical discrete semantic entropy (DSE) estimator, finding that it underestimates the "true" semantic entropy, as expected from theory. We propose a modified semantic alphabet size estimator, and illustrate that using it to adjust DSE for sample coverage results in more accurate SE estimation in our setting of interest. Furthermore, we find that two semantic alphabet size estimators, including our proposed, flag incorrect LLM responses as well or better than many top-performing alternatives, with the added benefit of remaining highly interpretable.

翻译：许多用于量化大语言模型（LLM）不确定性的黑盒技术依赖于重复的LLM采样，这可能导致高昂的计算开销。因此，实际应用要求能够基于少量样本进行可靠估计。语义熵（SE）是一种流行的基于样本的不确定性估计器，其离散化形式在黑盒设置中具有吸引力。SE的最新扩展在LLM幻觉检测方面表现出改进，但采用了可解释性较低的方法并引入了额外的超参数。为此，我们重新审视经典的离散语义熵（DSE）估计器，发现其低估了理论预期的“真实”语义熵。我们提出了一种改进的语义字母表规模估计器，并证明在目标场景中，利用该估计器对DSE进行样本覆盖调整可获得更准确的SE估计。此外，我们发现两种语义字母表规模估计器（包括我们提出的方法）在识别错误LLM响应方面表现与多数顶尖替代方法相当或更优，同时保持了高度可解释性的优势。