This paper deals with uncertainty quantification and out-of-distribution detection in deep learning using Bayesian and ensemble methods. It proposes a practical solution to the lack of prediction diversity observed recently for standard approaches when used out-of-distribution (Ovadia et al., 2019; Liu et al., 2021). Considering that this issue is mainly related to a lack of weight diversity, we claim that standard methods sample in "over-restricted" regions of the weight space due to the use of "over-regularization" processes, such as weight decay and zero-mean centered Gaussian priors. We propose to solve the problem by adopting the maximum entropy principle for the weight distribution, with the underlying idea to maximize the weight diversity. Under this paradigm, the epistemic uncertainty is described by the weight distribution of maximal entropy that produces neural networks "consistent" with the training observations. Considering stochastic neural networks, a practical optimization is derived to build such a distribution, defined as a trade-off between the average empirical risk and the weight distribution entropy. We develop a novel weight parameterization for the stochastic model, based on the singular value decomposition of the neural network's hidden representations, which enables a large increase of the weight entropy for a small empirical risk penalization. We provide both theoretical and numerical results to assess the efficiency of the approach. In particular, the proposed algorithm appears in the top three best methods in all configurations of an extensive out-of-distribution detection benchmark including more than thirty competitors.

翻译：本文探讨了利用贝叶斯方法和集成方法在深度学习中进行不确定性量化与分布外检测问题。针对近期标准方法在分布外场景下缺乏预测多样性（Ovadia 等，2019；Liu 等，2021），我们提出一种实用解决方案。考虑到该问题主要源于权重多样性不足，我们指出标准方法因采用“过度正则化”过程（如权重衰减和零均值高斯先验）而在权重空间“过度受限”区域进行采样。为此，我们提出基于权重分布的最大熵原理来解决问题，核心思想是最大化权重多样性。在该范式下，认知不确定性由最大熵权重分布描述，该分布产生的神经网络与训练观测“一致”。针对随机神经网络，我们推导出一种实用优化方法构建此类分布，将其定义为平均经验风险与权重分布熵之间的权衡。为随机模型开发了基于神经网络隐表示奇异值分解的新型权重参数化方法，使得在较小经验风险惩罚下大幅提升权重熵。我们提供理论分析与数值结果验证方法有效性，特别地，在包含三十余种方法的广泛分布外检测基准测试所有配置中，所提算法均位列前三。