Real-time 3D reconstruction of surgical scenes plays a vital role in computer-assisted surgery, holding a promise to enhance surgeons' visibility. Recent advancements in 3D Gaussian Splatting (3DGS) have shown great potential for real-time novel view synthesis of general scenes, which relies on accurate poses and point clouds generated by Structure-from-Motion (SfM) for initialization. However, 3DGS with SfM fails to recover accurate camera poses and geometry in surgical scenes due to the challenges of minimal textures and photometric inconsistencies. To tackle this problem, in this paper, we propose the first SfM-free 3DGS-based method for surgical scene reconstruction by jointly optimizing the camera poses and scene representation. Based on the video continuity, the key of our method is to exploit the immediate optical flow priors to guide the projection flow derived from 3D Gaussians. Unlike most previous methods relying on photometric loss only, we formulate the pose estimation problem as minimizing the flow loss between the projection flow and optical flow. A consistency check is further introduced to filter the flow outliers by detecting the rigid and reliable points that satisfy the epipolar geometry. During 3D Gaussian optimization, we randomly sample frames to optimize the scene representations to grow the 3D Gaussian progressively. Experiments on the SCARED dataset demonstrate our superior performance over existing methods in novel view synthesis and pose estimation with high efficiency. Code is available at https://github.com/wrld/Free-SurGS.

翻译：手术场景的实时三维重建在计算机辅助手术中起着至关重要的作用，有望增强外科医生的视野。三维高斯溅射（3DGS）的最新进展在通用场景的实时新视角合成方面展现出巨大潜力，其初始化依赖于运动恢复结构（SfM）生成的精确位姿和点云。然而，由于手术场景中纹理稀少和光度不一致的挑战，依赖SfM的3DGS方法无法准确恢复相机位姿和几何结构。为解决此问题，本文首次提出一种基于3DGS的无SfM手术场景重建方法，通过联合优化相机位姿与场景表示来实现。基于视频连续性，本方法的核心在于利用即时光流先验来引导从三维高斯导出的投影流。与大多数先前仅依赖光度损失的方法不同，我们将位姿估计问题表述为最小化投影流与光流之间的流损失。进一步引入一致性检查，通过检测满足对极几何的刚性和可靠点来过滤流异常值。在三维高斯优化过程中，我们随机采样帧以优化场景表示，从而逐步增长三维高斯。在SCARED数据集上的实验表明，我们的方法在新视角合成和位姿估计方面以高效率超越了现有方法。代码发布于 https://github.com/wrld/Free-SurGS。