The Laplace approximation provides a closed-form model selection objective for neural networks (NN). Online variants, which optimise NN parameters jointly with hyperparameters, like weight decay strength, have seen renewed interest in the Bayesian deep learning community. However, these methods violate Laplace's method's critical assumption that the approximation is performed around a mode of the loss, calling into question their soundness. This work re-derives online Laplace methods, showing them to target a variational bound on a mode-corrected variant of the Laplace evidence which does not make stationarity assumptions. Online Laplace and its mode-corrected counterpart share stationary points where 1. the NN parameters are a maximum a posteriori, satisfying the Laplace method's assumption, and 2. the hyperparameters maximise the Laplace evidence, motivating online methods. We demonstrate that these optima are roughly attained in practise by online algorithms using full-batch gradient descent on UCI regression datasets. The optimised hyperparameters prevent overfitting and outperform validation-based early stopping.

翻译：拉普拉斯近似为神经网络（NN）提供了一种闭式模型选择目标函数。在线变体在贝叶斯深度学习社区中重新受到关注，它能联合优化NN参数与超参数（如权重衰减强度）。然而，这些方法违反了拉普拉斯方法的关键假设——近似需在损失模态附近进行，这引发了对其合理性的质疑。本文重新推导了在线拉普拉斯方法，证明其目标是对拉普拉斯证据的模态校正变体施加变分界，无需平稳性假设。在线拉普拉斯及其模态校正对应物共享平稳点，其中：1.NN参数为最大后验估计，满足拉普拉斯方法的假设；2.超参数最大化拉普拉斯证据，为在线方法提供了动机。我们通过UCI回归数据集上的全批量梯度下降在线算法实证表明，这些最优解在实践中可近似达到。优化后的超参数能防止过拟合，且性能优于基于验证的早停法。