Stochastic gradient Langevin dynamics and its variants approximate the likelihood of an entire dataset, via random (and typically much smaller) subsets, in the setting of Bayesian sampling. Due to the (often substantial) improvement of the computational efficiency, they have been widely used in large-scale machine learning applications. It has been demonstrated that the so-called covariance-controlled adaptive Langevin (CCAdL) thermostat, which incorporates an additional term involving the covariance matrix of the noisy force, outperforms popular alternative methods. A moving average is used in CCAdL to estimate the covariance matrix of the noisy force, in which case the covariance matrix will converge to a constant matrix in long-time limit. Moreover, it appears in our numerical experiments that the use of a moving average could reduce the stability of the numerical integrators, thereby limiting the largest usable stepsize. In this article, we propose a modified CCAdL (i.e., mCCAdL) thermostat that uses the scaling part of the scaling and squaring method together with a truncated Taylor series approximation to the exponential to numerically approximate the exact solution to the subsystem involving the additional term proposed in CCAdL. We also propose a symmetric splitting method for mCCAdL, instead of an Euler-type discretisation used in the original CCAdL thermostat. We demonstrate in our numerical experiments that the newly proposed mCCAdL thermostat achieves a substantial improvement in the numerical stability over the original CCAdL thermostat, while significantly outperforming popular alternative stochastic gradient methods in terms of the numerical accuracy for large-scale machine learning applications.

翻译：随机梯度朗之万动力学及其变体在贝叶斯采样框架下，通过随机（且通常规模小得多）的数据子集来近似整个数据集的似然。由于其在计算效率上（常常是显著的）提升，这些方法已被广泛应用于大规模机器学习任务中。研究表明，所谓的协方差控制自适应朗之万恒温器通过引入一个包含噪声力协方差矩阵的附加项，其性能优于其他流行方法。CCAdL使用移动平均来估计噪声力的协方差矩阵，在这种情况下，协方差矩阵将在长时间极限下收敛到一个常数矩阵。此外，我们的数值实验表明，使用移动平均可能会降低数值积分器的稳定性，从而限制了最大可用步长。本文中，我们提出了一种改进的CCAdL恒温器，即mCCAdL。该方法利用缩放与平方法的缩放部分，结合指数函数的截断泰勒级数近似，来数值逼近CCAdL中提出的、包含附加项的子系统的精确解。我们还为mCCAdL提出了一种对称分裂方法，以替代原始CCAdL恒温器中使用的欧拉型离散化方案。我们的数值实验证明，新提出的mCCAdL恒温器在数值稳定性上相比原始CCAdL恒温器有显著提升，同时在大规模机器学习应用中，其数值精度也显著优于其他流行的随机梯度方法。