In supervised learning, understanding an input's proximity to the training data can help a model decide whether it has sufficient evidence for reaching a reliable prediction. While powerful probabilistic models such as Gaussian Processes naturally have this property, deep neural networks often lack it. In this paper, we introduce Distance Aware Bottleneck (DAB), i.e., a new method for enriching deep neural networks with this property. Building on prior information bottleneck approaches, our method learns a codebook that stores a compressed representation of all inputs seen during training. The distance of a new example from this codebook can serve as an uncertainty estimate for the example. The resulting model is simple to train and provides deterministic uncertainty estimates by a single forward pass. Finally, our method achieves better out-of-distribution (OOD) detection and misclassification prediction than prior methods, including expensive ensemble methods, deep kernel Gaussian Processes, and approaches based on the standard information bottleneck.

翻译：在监督学习中，理解输入样本与训练数据的邻近度有助于模型判断是否已获得足够证据以做出可靠预测。尽管高斯过程等强大的概率模型天然具备此特性，深度神经网络往往缺乏这一能力。本文提出距离感知瓶颈（DAB）方法，旨在为深度神经网络赋予此种特性。该方法基于先验信息瓶颈理论，通过学习一个存储训练期间所见全部输入压缩表示的码本。新样本与此码本的距离可作为该样本的不确定性估计。所得模型训练简便，仅需单次前向传播即可获得确定性不确定性估计。最终，本方法在分布外（OOD）检测与误分类预测任务上，超越了包括计算昂贵的集成方法、深度核高斯过程以及基于标准信息瓶颈方法在内的现有技术。