Uncertainty learning and quantification of models are crucial tasks to enhance the trustworthiness of the models. Importantly, the recent surge of generative language models (GLMs) emphasizes the need for reliable uncertainty quantification due to the concerns on generating hallucinated facts. In this paper, we propose to learn neural prediction set models that comes with the probably approximately correct (PAC) guarantee for quantifying the uncertainty of GLMs. Unlike existing prediction set models, which are parameterized by a scalar value, we propose to parameterize prediction sets via neural networks, which achieves more precise uncertainty quantification but still satisfies the PAC guarantee. We demonstrate the efficacy of our method on four types of language datasets and six types of models by showing that our method improves the quantified uncertainty by $63\%$ on average, compared to a standard baseline method.

翻译：不确定性学习与模型量化是增强模型可信度的关键任务。近年来，生成语言模型（GLMs）的快速发展引发了对幻觉事实生成问题的担忧，从而凸显了可靠不确定性量化的必要性。本文提出学习神经预测集模型，该模型具备可能近似正确（PAC）保证，可用于量化GLMs的不确定性。与现有通过标量参数化的预测集模型不同，我们提出通过神经网络参数化预测集，该方法在实现更精确不确定性量化的同时仍满足PAC保证。通过在四种语言数据集和六种模型上的实验，我们证明该方法相较标准基线方法平均可将量化不确定性提升63%。