Neural 3D representations such as Neural Radiance Fields (NeRF), excel at producing photo-realistic rendering results but lack the flexibility for manipulation and editing which is crucial for content creation. Previous works have attempted to address this issue by deforming a NeRF in canonical space or manipulating the radiance field based on an explicit mesh. However, manipulating NeRF is not highly controllable and requires a long training and inference time. With the emergence of 3D Gaussian Splatting (3DGS), extremely high-fidelity novel view synthesis can be achieved using an explicit point-based 3D representation with much faster training and rendering speed. However, there is still a lack of effective means to manipulate 3DGS freely while maintaining rendering quality. In this work, we aim to tackle the challenge of achieving manipulable photo-realistic rendering. We propose to utilize a triangular mesh to manipulate 3DGS directly with self-adaptation. This approach reduces the need to design various algorithms for different types of Gaussian manipulation. By utilizing a triangle shape-aware Gaussian binding and adapting method, we can achieve 3DGS manipulation and preserve high-fidelity rendering after manipulation. Our approach is capable of handling large deformations, local manipulations, and soft body simulations while keeping high-quality rendering. Furthermore, we demonstrate that our method is also effective with inaccurate meshes extracted from 3DGS. Experiments conducted demonstrate the effectiveness of our method and its superiority over baseline approaches.

翻译：神经三维表示（如神经辐射场NeRF）在生成逼真渲染结果方面表现出色，但缺乏对内容创作至关重要的灵活操控与编辑能力。先前研究尝试通过在规范空间变形NeRF或基于显式网格操控辐射场来解决此问题，但NeRF的操控性仍不够理想，且需要较长的训练与推理时间。随着三维高斯溅射（3DGS）技术的出现，基于显式点表示的三维模型能以更快的训练和渲染速度实现极高保真度的新视角合成。然而，目前仍缺乏在保持渲染质量的同时自由操控3DGS的有效手段。本研究致力于解决可操控逼真渲染的挑战，提出直接利用三角网格通过自适应机制操控3DGS。该方法无需为不同类型的高斯操作设计多种算法，通过三角形形状感知的高斯绑定与自适应方法，既能实现3DGS操控，又能在操控后保持高保真度渲染。我们的方法能够处理大尺度形变、局部操控及软体模拟，同时维持高质量渲染效果。此外，实验证明该方法对从3DGS提取的不精确网格同样有效。实验结果验证了本方法的有效性及其相对于基线方法的优越性。