3D Gaussian Splatting (3DGS) is widely used for novel view synthesis due to its high rendering quality and fast inference time. However, 3DGS predominantly relies on first-order optimizers such as Adam, which leads to long training times. To address this limitation, we propose a novel second-order optimization strategy based on Levenberg-Marquardt (LM) and Conjugate Gradient (CG), which we specifically tailor towards Gaussian Splatting. Our key insight is that the Jacobian in 3DGS exhibits significant sparsity since each Gaussian affects only a limited number of pixels. We exploit this sparsity by proposing a matrix-free and GPU-parallelized LM optimization. To further improve its efficiency, we propose sampling strategies for both the camera views and loss function and, consequently, the normal equation, significantly reducing the computational complexity. In addition, we increase the convergence rate of the second-order approximation by introducing an effective heuristic to determine the learning rate that avoids the expensive computation cost of line search methods. As a result, our method achieves a $3\times$ speedup over standard LM and outperforms Adam by $~6\times$ when the Gaussian count is low while remaining competitive for moderate counts. Project Page: https://vcai.mpi-inf.mpg.de/projects/LM-IS

翻译：三维高斯溅射（3DGS）因其高渲染质量与快速推理能力，被广泛用于新视角合成。然而，3DGS主要依赖Adam等一阶优化器，导致训练时间较长。为克服此限制，本文提出一种基于Levenberg-Marquardt（LM）与共轭梯度（CG）的新型二阶优化策略，并专门针对高斯溅射特性进行定制。我们的核心发现是：由于每个高斯仅影响有限数量的像素，3DGS中的雅可比矩阵具有显著稀疏性。通过提出矩阵无关且GPU并行化的LM优化方法，我们充分利用了这种稀疏特性。为进一步提升效率，我们针对相机视角、损失函数及相应的正规方程提出采样策略，显著降低了计算复杂度。此外，通过引入一种确定学习率的有效启发式方法（避免线搜索方法的高昂计算成本），我们提升了二阶近似的收敛速度。实验表明，本方法在低高斯数量情况下较标准LM实现3倍加速，较Adam提升约6倍速度，在中规模高斯数量下仍保持竞争力。项目页面：https://vcai.mpi-inf.mpg.de/projects/LM-IS