3D reconstruction and simulation, although interrelated, have distinct objectives: reconstruction requires a flexible 3D representation that can adapt to diverse scenes, while simulation needs a structured representation to model motion principles effectively. This paper introduces the Mesh-adsorbed Gaussian Splatting (MaGS) method to address this challenge. MaGS constrains 3D Gaussians to roam near the mesh, creating a mutually adsorbed mesh-Gaussian 3D representation. Such representation harnesses both the rendering flexibility of 3D Gaussians and the structured property of meshes. To achieve this, we introduce RMD-Net, a network that learns motion priors from video data to refine mesh deformations, alongside RGD-Net, which models the relative displacement between the mesh and Gaussians to enhance rendering fidelity under mesh constraints. To generalize to novel, user-defined deformations beyond input video without reliance on temporal data, we propose MPE-Net, which leverages inherent mesh information to bootstrap RMD-Net and RGD-Net. Due to the universality of meshes, MaGS is compatible with various deformation priors such as ARAP, SMPL, and soft physics simulation. Extensive experiments on the D-NeRF, DG-Mesh, and PeopleSnapshot datasets demonstrate that MaGS achieves state-of-the-art performance in both reconstruction and simulation.

翻译：三维重建与仿真虽相互关联，却具有不同目标：重建需要能适应多样化场景的灵活三维表征，而仿真则需要结构化表征以有效建模运动规律。本文提出网格吸附高斯溅射（MaGS）方法以应对这一挑战。MaGS将三维高斯约束在网格近邻区域游走，构建出相互吸附的网格-高斯三维表征。该表征同时利用了三维高斯在渲染上的灵活性与网格的结构化特性。为实现这一目标，我们引入了RMD-Net——一个从视频数据学习运动先验以优化网格变形的网络，以及RGD-Net——该网络建模网格与高斯之间的相对位移，以在网格约束下提升渲染保真度。为将方法泛化至输入视频之外、用户自定义的新颖形变场景（无需依赖时序数据），我们提出了MPE-Net，该网络利用网格固有信息为RMD-Net与RGD-Net提供初始化支持。得益于网格的普适性，MaGS兼容多种形变先验模型，如ARAP、SMPL及软体物理仿真。在D-NeRF、DG-Mesh和PeopleSnapshot数据集上的大量实验表明，MaGS在重建与仿真任务上均达到了最先进的性能水平。