Deep neural networks (NNs) are known for their high-prediction performances. However, NNs are prone to yield unreliable predictions when encountering completely new situations without indicating their uncertainty. Bayesian variants of NNs (BNNs), such as Monte Carlo (MC) dropout BNNs, do provide uncertainty measures and simultaneously increase the prediction performance. The only disadvantage of BNNs is their higher computation time during test time because they rely on a sampling approach. Here we present a single-shot MC dropout approximation that preserves the advantages of BNNs while being as fast as NNs. Our approach is based on moment propagation (MP) and allows to analytically approximate the expected value and the variance of the MC dropout signal for commonly used layers in NNs, i.e. convolution, max pooling, dense, softmax, and dropout layers. The MP approach can convert an NN into a BNN without re-training given the NN has been trained with standard dropout. We evaluate our approach on different benchmark datasets and a simulated toy example in a classification and regression setting. We demonstrate that our single-shot MC dropout approximation resembles the point estimate and the uncertainty estimate of the predictive distribution that is achieved with an MC approach, while being fast enough for real-time deployments of BNNs. We show that using part of the saved time to combine our MP approach with deep ensemble techniques does further improve the uncertainty measures.

翻译：深度神经网络（NNs）以高预测性能著称，但在遭遇全新情境时易产生不可靠预测且无法表征其不确定性。神经网络的贝叶斯变体（BNNs），如蒙特卡洛（MC）dropout BNNs，既能提供不确定性度量又能提升预测性能。BNNs的唯一缺点是测试阶段计算时间较长，因其依赖采样方法。本文提出一种单次MC dropout近似方法，在保持BNNs优势的同时实现与NNs相当的计算速度。该方法基于矩传播（MP）技术，能够解析近似NNs常用层（包括卷积层、最大池化层、全连接层、softmax层和dropout层）中MC dropout信号的期望值和方差。对于已通过标准dropout训练的NNs，MP方法无需重新训练即可将其转化为BNN。我们在不同基准数据集及模拟玩具示例的分类与回归场景中验证该方法。实验表明，本研究的单次MC dropout近似方法既能复现MC方法所得预测分布的点估计与不确定性估计，同时其运算速度足以满足BNNs的实时部署需求。此外，我们将节省的部分计算时间用于MP方法与深度集成技术的结合，进一步提升了不确定性度量的效果。