Accurate 3D tracking in highly deformable scenes with occlusions and shadows can facilitate new applications in robotics, augmented reality, and generative AI. However, tracking under these conditions is extremely challenging due to the ambiguity that arises with large deformations, shadows, and occlusions. We introduce MD-Splatting, an approach for simultaneous 3D tracking and novel view synthesis, using video captures of a dynamic scene from various camera poses. MD-Splatting builds on recent advances in Gaussian splatting, a method that learns the properties of a large number of Gaussians for state-of-the-art and fast novel view synthesis. MD-Splatting learns a deformation function to project a set of Gaussians with non-metric, thus canonical, properties into metric space. The deformation function uses a neural-voxel encoding and a multilayer perceptron (MLP) to infer Gaussian position, rotation, and a shadow scalar. We enforce physics-inspired regularization terms based on local rigidity, conservation of momentum, and isometry, which leads to trajectories with smaller trajectory errors. MD-Splatting achieves high-quality 3D tracking on highly deformable scenes with shadows and occlusions. Compared to state-of-the-art, we improve 3D tracking by an average of 23.9 %, while simultaneously achieving high-quality novel view synthesis. With sufficient texture such as in scene 6, MD-Splatting achieves a median tracking error of 3.39 mm on a cloth of 1 x 1 meters in size. Project website: https://md-splatting.github.io/.

翻译：我们提出MD-Splatting方法，用于在存在遮挡与阴影的高形变场景中实现同时三维追踪与新颖视角合成，该方法利用不同相机位姿下动态场景的视频捕捉数据。MD-Splatting建立在近期高斯溅射（Gaussian splatting）研究进展的基础上——该技术通过学习大量高斯属性实现快速且高质量的新颖视角合成。MD-Splatting学习形变函数，将具有非度量（即标准）属性的一组高斯投影到度量空间。该形变函数采用神经体素编码与多层感知机（MLP）来推断高斯位置、旋转和平移标量。我们基于局部刚性、动量守恒与等距性引入物理启发式正则项，从而显著降低轨迹误差。MD-Splatting在存在阴影与遮挡的高形变场景中实现高质量三维追踪，相较于现有最优方法平均提升23.9%的三维追踪精度，同时保持高质量新颖视角合成。在具有充足纹理的场景（如场景6）中，MD-Splatting对1×1米布料的中位数追踪误差仅为3.39毫米。项目网站：https://md-splatting.github.io/。