Accurately quantifying a large language model's (LLM) predictive uncertainty is crucial for judging the reliability of its answers. While most existing research focuses on short, directly answerable questions with closed-form outputs (e.g., multiple-choice), involving intermediate reasoning steps in LLM responses is increasingly important. This added complexity complicates uncertainty quantification (UQ) because the probabilities assigned to answer tokens are conditioned on a vast space of preceding reasoning tokens. Direct marginalization is infeasible, and the dependency inflates probability estimates, causing overconfidence in UQ. To address this, we propose UQAC, an efficient method that narrows the reasoning space to a tractable size for marginalization. UQAC iteratively constructs an "attention chain" of tokens deemed "semantically crucial" to the final answer via a backtracking procedure. Starting from the answer tokens, it uses attention weights to identify the most influential predecessors, then iterates this process until reaching the input tokens. Similarity filtering and probability thresholding further refine the resulting chain, allowing us to approximate the marginal probabilities of the answer tokens, which serve as the LLM's confidence. We validate UQAC on multiple reasoning benchmarks with advanced open-source LLMs, demonstrating that it consistently delivers reliable UQ estimates with high computational efficiency.

翻译：准确量化大型语言模型（LLM）的预测不确定性，对于判断其答案的可靠性至关重要。现有研究大多集中于具有封闭形式输出（如多项选择）的简短、可直接回答的问题，然而在LLM响应中引入中间推理步骤正变得越来越重要。这种增加的复杂性使得不确定性量化（UQ）变得困难，因为分配给答案词元的概率是以海量的先前推理词元空间为条件的。直接进行边缘化计算不可行，且这种依赖性会抬高概率估计值，导致UQ中的过度自信。为解决此问题，我们提出了UQAC，一种通过将推理空间缩小到可处理规模以进行边缘化的高效方法。UQAC通过回溯过程，迭代式地构建一个被视为对最终答案“语义关键”的词元“注意力链”。该方法从答案词元开始，利用注意力权重识别最具影响力的前驱词元，然后迭代此过程直至到达输入词元。相似性过滤和概率阈值处理进一步精炼所得的链，从而使我们能够近似计算答案词元的边缘概率，以此作为LLM的置信度。我们在多个推理基准上使用先进的开源LLM验证了UQAC，结果表明它能够以高计算效率持续提供可靠的UQ估计。