The Laplace approximation provides a closed-form model selection objective for neural networks (NN). Online variants, which optimise NN parameters jointly with hyperparameters, like weight decay strength, have seen renewed interest in the Bayesian deep learning community. However, these methods violate Laplace's method's critical assumption that the approximation is performed around a mode of the loss, calling into question their soundness. This work re-derives online Laplace methods, showing them to target a variational bound on a mode-corrected variant of the Laplace evidence which does not make stationarity assumptions. Online Laplace and its mode-corrected counterpart share stationary points where 1. the NN parameters are a maximum a posteriori, satisfying the Laplace method's assumption, and 2. the hyperparameters maximise the Laplace evidence, motivating online methods. We demonstrate that these optima are roughly attained in practise by online algorithms using full-batch gradient descent on UCI regression datasets. The optimised hyperparameters prevent overfitting and outperform validation-based early stopping.

翻译：拉普拉斯近似为神经网络（NN）提供了一种闭式的模型选择目标函数。在线变体方法联合优化神经网络参数与超参数（如权重衰减强度），在贝叶斯深度学习领域重新引起了研究兴趣。然而，这些方法违反了拉普拉斯方法的关键假设——近似必须在损失模态附近进行，这对其理论合理性提出了质疑。本文重新推导了在线拉普拉斯方法，证明其目标是对拉普拉斯证据的模态校正变体进行变分界优化，且无需平稳性假设。在线拉普拉斯方法及其模态校正变体共享稳定点，在这些稳定点处：1. 神经网络参数满足最大后验估计，符合拉普拉斯方法的假设；2. 超参数最大化拉普拉斯证据，从而为在线方法提供了理论依据。我们通过UCI回归数据集上的全批量梯度下降在线算法实验证明，这些最优解在实际中近似可达。优化后的超参数可有效防止过拟合，且性能优于基于验证集的早停策略。