Calibrating language models (LMs) aligns their generation confidence with the actual likelihood of answer correctness, which can inform users about LMs' reliability and mitigate hallucinated content. However, prior calibration methods, such as self-consistency-based and logit-based approaches, are either limited in inference-time efficiency or fall short of providing informative signals. Moreover, simply filtering out low-confidence responses reduces the LM's helpfulness when the answers are correct. Therefore, effectively using calibration techniques to enhance an LM's factuality remains an unsolved challenge. In this paper, we first propose an activation-based calibration method, ActCab, which trains a linear layer on top of the LM's last-layer activations that can better capture the representations of knowledge. Built on top of ActCab, we further propose CoDec, a confidence-guided decoding strategy to elicit truthful answers with high confidence from LMs. By evaluating on five popular QA benchmarks, ActCab achieves superior calibration performance than all competitive baselines, e.g., by reducing the average expected calibration error (ECE) score by up to 39%. Further experiments on CoDec show consistent improvements in several LMs' factuality on challenging QA datasets, such as TruthfulQA, highlighting the value of confidence signals in enhancing factuality.

翻译：校准语言模型（LMs）旨在使其生成置信度与答案正确性的实际可能性对齐，这既能向用户提示LMs的可靠性，也能减少幻觉内容的产生。然而，现有的校准方法（如基于自一致性和基于逻辑值的方法）要么受限于推理效率，要么无法提供足够的信息量信号。此外，简单过滤低置信度响应会在答案正确时降低LM的辅助效用。因此，如何有效利用校准技术提升LM的事实性仍是一个未解决的挑战。本文首先提出一种基于激活的校准方法ActCab，该方法在LM最后一层激活值上训练一个线性层，以更好地捕捉知识表征。基于ActCab，我们进一步提出CoDec——一种置信度引导的解码策略，旨在从LMs中引导出高置信度的真实答案。通过在五个常用QA基准上的评估，ActCab实现了优于所有竞争基线的校准性能，例如将平均期望校准误差（ECE）分数降低达39%。对CoDec的进一步实验表明，在TruthfulQA等具有挑战性的QA数据集上，多种LMs的事实性均获得持续提升，这凸显了置信度信号在增强事实性方面的价值。