The Linearized Laplace Approximation (LLA) has been recently used to perform uncertainty estimation on the predictions of pre-trained deep neural networks (DNNs). However, its widespread application is hindered by significant computational costs, particularly in scenarios with a large number of training points or DNN parameters. Consequently, additional approximations of LLA, such as Kronecker-factored or diagonal approximate GGN matrices, are utilized, potentially compromising the model's performance. To address these challenges, we propose a new method for approximating LLA using a variational sparse Gaussian Process (GP). Our method is based on the dual RKHS formulation of GPs and retains, as the predictive mean, the output of the original DNN. Furthermore, it allows for efficient stochastic optimization, which results in sub-linear training time in the size of the training dataset. Specifically, its training cost is independent of the number of training points. We compare our proposed method against accelerated LLA (ELLA), which relies on the Nystr\"om approximation, as well as other LLA variants employing the sample-then-optimize principle. Experimental results, both on regression and classification datasets, show that our method outperforms these already existing efficient variants of LLA, both in terms of the quality of the predictive distribution and in terms of total computational time.

翻译：线性化拉普拉斯近似（LLA）近年来被用于对预训练深度神经网络（DNNs）的预测进行不确定性估计。然而，其广泛应用受到显著计算成本的阻碍，特别是在训练点数量或DNN参数规模较大的场景中。因此，通常采用LLA的额外近似方法，例如Kronecker分解或对角近似GGN矩阵，但这可能损害模型的性能。为应对这些挑战，我们提出了一种使用变分稀疏高斯过程（GP）来近似LLA的新方法。我们的方法基于GP的对偶RKHS（再生核希尔伯特空间）公式，并将原始DNN的输出保留为预测均值。此外，该方法支持高效的随机优化，从而在训练数据集规模上实现亚线性训练时间。具体而言，其训练成本与训练点的数量无关。我们将所提出的方法与依赖Nyström近似的加速LLA（ELLA），以及采用“采样-优化”原则的其他LLA变体进行了比较。在回归和分类数据集上的实验结果表明，无论是在预测分布的质量方面，还是在总计算时间方面，我们的方法均优于这些已有的高效LLA变体。