Since training deep neural networks takes significant computational resources, extending the training dataset with new data is difficult, as it typically requires complete retraining. Moreover, specific applications do not allow costly retraining due to time or computational constraints. We address this issue by proposing a novel Bayesian update method for deep neural networks by using a last-layer Laplace approximation. Concretely, we leverage second-order optimization techniques on the Gaussian posterior distribution of a Laplace approximation, computing the inverse Hessian matrix in closed form. This way, our method allows for fast and effective updates upon the arrival of new data in a stationary setting. A large-scale evaluation study across different data modalities confirms that our updates are a fast and competitive alternative to costly retraining. Furthermore, we demonstrate its applicability in a deep active learning scenario by using our update to improve existing selection strategies.

翻译：由于训练深度神经网络需要大量计算资源，使用新数据扩展训练集通常十分困难，因为这往往需要完整的重新训练。此外，特定应用场景因时间或计算资源限制无法承担昂贵的重新训练成本。针对这一问题，我们提出一种基于最后一层拉普拉斯近似的新型深度神经网络贝叶斯更新方法。具体而言，我们在拉普拉斯近似的高斯后验分布上应用二阶优化技术，以闭式形式计算逆海森矩阵。通过这种方式，本方法能够在平稳环境下快速有效地实现新数据到达后的模型更新。跨不同数据模态的大规模评估研究表明，我们的更新方法是替代昂贵重新训练的一种快速且具有竞争力的方案。此外，我们通过在深度主动学习场景中应用本更新方法来改进现有选择策略，验证了其实际适用性。