Uncertainty quantification of neural networks is critical to measuring the reliability and robustness of deep learning systems. However, this often involves costly or inaccurate sampling methods and approximations. This paper presents a sample-free moment propagation technique that propagates mean vectors and covariance matrices across a network to accurately characterize the input-output distributions of neural networks. A key enabler of our technique is an analytic solution for the covariance of random variables passed through nonlinear activation functions, such as Heaviside, ReLU, and GELU. The wide applicability and merits of the proposed technique are shown in experiments analyzing the input-output distributions of trained neural networks and training Bayesian neural networks.

翻译：神经网络的不确定性量化对于衡量深度学习系统的可靠性和鲁棒性至关重要。然而，这通常涉及代价高昂或精度有限的采样方法与近似处理。本文提出一种免采样的矩传播技术，通过在全网络范围内传播均值向量和协方差矩阵，精确刻画神经网络的输入-输出分布。该技术的核心突破在于：为经过非线性激活函数（如Heaviside、ReLU和GELU）的随机变量协方差推导出解析解。通过在分析已训练神经网络输入-输出分布及训练贝叶斯神经网络等实验中的验证，我们展示了该技术的广泛适用性与显著优势。