Accurate confidence estimation is essential for trustworthy large language models (LLMs) systems, as it empowers the user to determine when to trust outputs and enables reliable deployment in safety-critical applications. Current confidence estimation methods for LLMs neglect the relevance between responses and contextual information, a crucial factor in output quality evaluation, particularly in scenarios where background knowledge is provided. To bridge this gap, we propose CRUX (Context-aware entropy Reduction and Unified consistency eXamination), the first framework that integrates context faithfulness and consistency for confidence estimation via two novel metrics. First, contextual entropy reduction represents data uncertainty with the information gain through contrastive sampling with and without context. Second, unified consistency examination captures potential model uncertainty through the global consistency of the generated answers with and without context. Experiments across three benchmark datasets (CoQA, SQuAD, QuAC) and two domain-specific datasets (BioASQ, EduQG) demonstrate CRUX's effectiveness, achieving the highest AUROC than existing baselines.

翻译：精确的置信度估计对于可信赖的大语言模型（LLM）系统至关重要，它使用户能够判断何时信任模型输出，并确保在安全关键应用中的可靠部署。当前针对LLM的置信度估计方法忽略了响应与情境信息之间的相关性，而这是评估输出质量的关键因素，尤其是在提供背景知识的应用场景中。为弥补这一不足，我们提出了CRUX（情境感知熵减与统一一致性检验）框架，这是首个通过两个新颖指标整合情境忠实度与一致性进行置信度估计的框架。首先，情境熵减通过对比有无情境的对比采样，以信息增益表征数据不确定性。其次，统一一致性检验通过生成答案在有无情境下的全局一致性来捕捉潜在模型不确定性。在三个基准数据集（CoQA、SQuAD、QuAC）和两个领域特定数据集（BioASQ、EduQG）上的实验验证了CRUX的有效性，其AUROC指标均优于现有基线方法。