Low-precision training has emerged as a promising low-cost technique to enhance the training efficiency of deep neural networks without sacrificing much accuracy. Its Bayesian counterpart can further provide uncertainty quantification and improved generalization accuracy. This paper investigates low-precision sampling via Stochastic Gradient Hamiltonian Monte Carlo (SGHMC) with low-precision and full-precision gradient accumulators for both strongly log-concave and non-log-concave distributions. Theoretically, our results show that, to achieve $\epsilon$-error in the 2-Wasserstein distance for non-log-concave distributions, low-precision SGHMC achieves quadratic improvement ($\widetilde{\mathbf{O}}\left({\epsilon^{-2}{\mu^*}^{-2}\log^2\left({\epsilon^{-1}}\right)}\right)$) compared to the state-of-the-art low-precision sampler, Stochastic Gradient Langevin Dynamics (SGLD) ($\widetilde{\mathbf{O}}\left({{\epsilon}^{-4}{\lambda^{*}}^{-1}\log^5\left({\epsilon^{-1}}\right)}\right)$). Moreover, we prove that low-precision SGHMC is more robust to the quantization error compared to low-precision SGLD due to the robustness of the momentum-based update w.r.t. gradient noise. Empirically, we conduct experiments on synthetic data, and {MNIST, CIFAR-10 \& CIFAR-100} datasets, which validate our theoretical findings. Our study highlights the potential of low-precision SGHMC as an efficient and accurate sampling method for large-scale and resource-limited machine learning.

翻译：低精度训练已成为一种有前景的低成本技术，可在不损失太多精度的情况下提升深度神经网络的训练效率。其贝叶斯变体能够进一步提供不确定性量化和改进的泛化精度。本文研究了基于低精度和全精度梯度累积器的随机梯度哈密尔顿蒙特卡洛方法（SGHMC）在强对数凹分布和非对数凹分布下的低精度采样。理论上，我们的结果表明，为实现非对数凹分布在2-瓦瑟斯坦距离下的ϵ误差，低精度SGHMC相比当前最优的低精度采样器——随机梯度朗之万动力学（SGLD）（其复杂度为$\widetilde{\mathbf{O}}\left({{\epsilon}^{-4}{\lambda^{*}}^{-1}\log^5\left({\epsilon^{-1}}\right)}\right)$），实现了二次改进（$\widetilde{\mathbf{O}}\left({\epsilon^{-2}{\mu^*}^{-2}\log^2\left({\epsilon^{-1}}\right)}\right)$）。此外，我们证明由于基于动量的更新对梯度噪声具有鲁棒性，低精度SGHMC相比低精度SGLD对量化误差更具鲁棒性。在实验上，我们在合成数据以及{MNIST、CIFAR-10和CIFAR-100}数据集上进行了验证，结果证实了我们的理论发现。本研究凸显了低精度SGHMC作为大规模和资源受限机器学习场景中高效且精确的采样方法的潜力。