Deployed language models must decide not only what to answer but also when not to answer. We present UniCR, a unified framework that turns heterogeneous uncertainty evidence including sequence likelihoods, self-consistency dispersion, retrieval compatibility, and tool or verifier feedback into a calibrated probability of correctness and then enforces a user-specified error budget via principled refusal. UniCR learns a lightweight calibration head with temperature scaling and proper scoring, supports API-only models through black-box features, and offers distribution-free guarantees using conformal risk control. For long-form generation, we align confidence with semantic fidelity by supervising on atomic factuality scores derived from retrieved evidence, reducing confident hallucinations while preserving coverage. Experiments on short-form QA, code generation with execution tests, and retrieval-augmented long-form QA show consistent improvements in calibration metrics, lower area under the risk-coverage curve, and higher coverage at fixed risk compared to entropy or logit thresholds, post-hoc calibrators, and end-to-end selective baselines. Analyses reveal that evidence contradiction, semantic dispersion, and tool inconsistency are the dominant drivers of abstention, yielding informative user-facing refusal messages. The result is a portable recipe of evidence fusion to calibrated probability to risk-controlled decision that improves trustworthiness without fine-tuning the base model and remains valid under distribution shift.

翻译：部署的语言模型不仅需要决定回答什么，还要决定何时不应作答。我们提出UniCR，一个统一框架，可将序列似然度、自一致性离散度、检索兼容性以及工具或验证器反馈等异质性不确定性证据，转化为校准后的正确概率，并通过原则性拒答机制强制执行用户指定的误差预算。UniCR通过学习轻量级校准头（含温度缩放和适当评分函数），支持仅通过黑盒特征访问的API模型，并利用共形风险控制提供无分布假设的保障。针对长文本生成任务，我们通过监督基于检索证据推导的原子事实性评分，将置信度与语义保真度对齐，从而在保持覆盖率的同时减少自信幻觉。在短文本问答、含执行测试的代码生成以及检索增强型长文本问答上的实验表明，与熵值或对数阈值、事后校准器及端到端选择性基线方法相比，该方法在校准指标、风险-覆盖率曲线下面积以及固定风险下的覆盖率方面均取得一致性改进。分析表明，证据矛盾、语义离散度和工具不一致性是模型选择拒答的主要驱动因素，并由此生成具有信息量的面向用户的拒答消息。这套可迁移的证据融合→校准概率→风险控制决策方案，能够在不微调基础模型的前提下提升可信度，且在分布偏移场景下依然有效。