Optimization is often cast as a deterministic problem, where the solution is found through some iterative procedure such as gradient descent. However, when training neural networks the loss function changes over (iteration) time due to the randomized selection of a subset of the samples. This randomization turns the optimization problem into a stochastic one. We propose to consider the loss as a noisy observation with respect to some reference optimum. This interpretation of the loss allows us to adopt Kalman filtering as an optimizer, as its recursive formulation is designed to estimate unknown parameters from noisy measurements. Moreover, we show that the Kalman Filter dynamical model for the evolution of the unknown parameters can be used to capture the gradient dynamics of advanced methods such as Momentum and Adam. We call this stochastic optimization method KOALA, which is short for Kalman Optimization Algorithm with Loss Adaptivity. KOALA is an easy to implement, scalable, and efficient method to train neural networks. We provide convergence analysis and show experimentally that it yields parameter estimates that are on par with or better than existing state of the art optimization algorithms across several neural network architectures and machine learning tasks, such as computer vision and language modeling.

翻译：优化往往被作为一种确定性的问题, 其解决方案是通过一些迭代程序( 如梯度下降) 找到的。 然而, 当培训神经网络时, 随机选择样本中的某个子集, 随机化将优化问题转化为随机性的问题。 我们提议将损失视为对某种参考的最佳度的噪音观测。 对损失的解释使我们能够将卡尔曼过滤器当作优化器, 因为它的循环配方旨在估计噪音测量的未知参数。 此外, 我们显示, Kalman 过滤器的未知参数演变动态模型可以用来捕捉先进的方法( 如 Momentum 和 Adam ) 的梯度动态。 我们称之为 KOALA 。 这对于 Kal Opmantimination Algorithm 来说是短于某些参考度最佳的。 KOALA是一个容易执行、 缩放和高效的方法, 用于培训神经网络。 我们提供趋同分析, 实验性地显示, 它产生的参数估计值与现有的计算机模型和图像分析相比, 以及 各种机器的模型算算等结构的状态相同或更好。