The Laplace approximation (LA) of the Bayesian posterior is a Gaussian distribution centered at the maximum a posteriori estimate. Its appeal in Bayesian deep learning stems from the ability to quantify uncertainty post-hoc (i.e., after standard network parameter optimization), the ease of sampling from the approximate posterior, and the analytic form of model evidence. However, an important computational bottleneck of LA is the necessary step of calculating and inverting the Hessian matrix of the log posterior. The Hessian may be approximated in a variety of ways, with quality varying with a number of factors including the network, dataset, and inference task. In this paper, we propose an alternative framework that sidesteps Hessian calculation and inversion. The Hessian-free Laplace (HFL) approximation uses curvature of both the log posterior and network prediction to estimate its variance. Only two point estimates are needed: the standard maximum a posteriori parameter and the optimal parameter under a loss regularized by the network prediction. We show that, under standard assumptions of LA in Bayesian deep learning, HFL targets the same variance as LA, and can be efficiently amortized in a pre-trained network. Experiments demonstrate comparable performance to that of exact and approximate Hessians, with excellent coverage for in-between uncertainty.

翻译：拉普拉斯近似（LA）将贝叶斯后验表示为以最大后验估计为中心的多元高斯分布。其在贝叶斯深度学习中的吸引力源于：能够事后（即标准网络参数优化后）量化不确定性、易于从近似后验中采样，以及模型证据的解析形式。然而，LA的关键计算瓶颈在于必须计算并求逆对数后验的Hessian矩阵。Hessian矩阵可通过多种方式近似，其质量受网络结构、数据集和推理任务等多种因素影响。本文提出一种绕过Hessian计算与求逆的替代框架。免Hessian拉普拉斯近似（HFL）利用对数后验与网络预测二者的曲率来估计方差，仅需两个点估计值：标准最大后验参数，以及受网络预测正则化的损失函数下的最优参数。我们证明，在贝叶斯深度学习LA的标准假设下，HFL能逼近与LA相同的方差，并可高效地分摊至预训练网络。实验表明，其性能与精确或近似Hessian方法相当，且对中间状态不确定性具有优异覆盖能力。