3D Gaussian Splatting (3DGS) is widely used for novel view synthesis due to its high rendering quality and fast inference time. However, 3DGS predominantly relies on first-order optimizers such as Adam, which leads to long training times. To address this limitation, we propose a novel second-order optimization strategy based on Levenberg-Marquardt (LM) and Conjugate Gradient (CG), specifically tailored towards Gaussian Splatting. Our key insight is that the Jacobian in 3DGS exhibits significant sparsity since each Gaussian affects only a limited number of pixels. We exploit this sparsity by proposing a matrix-free and GPU-parallelized LM optimization. To further improve its efficiency, we propose sampling strategies for both camera views and loss function and, consequently, the normal equation, significantly reducing the computational complexity. In addition, we increase the convergence rate of the second-order approximation by introducing an effective heuristic to determine the learning rate that avoids the expensive computation cost of line search methods. As a result, our method achieves a 4x speedup over standard LM and outperforms Adam by ~5x when the Gaussian count is low while providing ~1.3x speed in moderate counts. In addition, our matrix-free implementation achieves 2x speedup over the concurrent second-order optimizer 3DGS-LM, while using 3.5x less memory. Project Page: https://vcai.mpi-inf.mpg.de/projects/LM-RS/

翻译：三维高斯溅射（3DGS）因其高渲染质量与快速推理能力，被广泛用于新视角合成。然而，3DGS主要依赖Adam等一阶优化器，导致训练时间较长。为克服此限制，本文提出一种专为高斯溅射设计的、基于Levenberg-Marquardt（LM）与共轭梯度（CG）的新型二阶优化策略。我们的核心洞见在于：由于每个高斯仅影响有限数量的像素，3DGS中的雅可比矩阵具有显著稀疏性。我们通过提出一种矩阵无约束且GPU并行化的LM优化方法来利用该稀疏性。为进一步提升效率，我们针对相机视角与损失函数（进而针对法方程）提出采样策略，显著降低了计算复杂度。此外，我们通过引入一种确定学习率的有效启发式方法（避免线搜索方法的高昂计算成本），提高了二阶近似的收敛速度。实验表明，本方法较标准LM实现4倍加速，在低高斯数量下较Adam提速约5倍，在中等数量下提速约1.3倍。同时，我们的矩阵无约束实现较同期二阶优化器3DGS-LM提速2倍，且内存占用减少3.5倍。项目页面：https://vcai.mpi-inf.mpg.de/projects/LM-RS/