Trustworthy prediction in Deep Neural Networks (DNNs), including Pre-trained Language Models (PLMs) is important for safety-critical applications in the real world. However, DNNs often suffer from uncertainty estimation, such as miscalibration. In particular, approaches that require multiple stochastic inference can mitigate this problem, but the expensive cost of inference makes them impractical. In this study, we propose $k$-Nearest Neighbor Uncertainty Estimation ($k$NN-UE), which is an uncertainty estimation method that uses the distances from the neighbors and label-existence ratio of neighbors. Experiments on sentiment analysis, natural language inference, and named entity recognition show that our proposed method outperforms the baselines or recent density-based methods in confidence calibration, selective prediction, and out-of-distribution detection. Moreover, our analyses indicate that introducing dimension reduction or approximate nearest neighbor search inspired by recent $k$NN-LM studies reduces the inference overhead without significantly degrading estimation performance when combined them appropriately.

翻译：深度神经网络（DNNs），包括预训练语言模型（PLMs）的可信预测对于现实世界中的安全关键应用至关重要。然而，DNNs 经常面临不确定性估计问题，例如校准错误。特别是，需要多次随机推理的方法可以缓解此问题，但高昂的推理成本使其不切实际。在本研究中，我们提出 $k$-最近邻不确定性估计（$k$NN-UE），这是一种利用最近邻距离和邻居标签存在比例的不确定性估计方法。在情感分析、自然语言推理和命名实体识别上的实验表明，我们提出的方法在置信度校准、选择性预测和分布外检测方面优于基线方法或最近的基于密度的方法。此外，我们的分析表明，借鉴近期 $k$NN-LM 研究引入的降维或近似最近邻搜索技术，在适当结合使用时，能够显著降低推理开销而不会明显降低估计性能。