Joint rendering and deformation of mesh and 3D Gaussian Splatting (3DGS) have significant value as both representa tions offer complementary advantages for graphics applica tions. However, due to differences in representation and ren dering pipelines, existing studies render meshes and 3DGS separately, making it difficult to accurately handle occlusions and transparency. Moreover, the deformed 3DGS still suffers from visual artifacts due to the sensitivity to the topology quality of the proxy mesh. These issues pose serious obsta cles to the joint use of 3DGS and meshes, making it diffi cult to adapt 3DGS to conventional mesh-oriented graphics pipelines. We propose UniMGS, the first unified framework for rasterizing mesh and 3DGS in a single-pass anti-aliased manner, with a novel binding strategy for 3DGS deformation based on proxy mesh. Our key insight is to blend the col ors of both triangle and Gaussian fragments by anti-aliased α-blending in a single pass, achieving visually coherent re sults with precise handling of occlusion and transparency. To improve the visual appearance of the deformed 3DGS, our Gaussian-centric binding strategy employs a proxy mesh and spatially associates Gaussians with the mesh faces, signifi cantly reducing rendering artifacts. With these two compo nents, UniMGS enables the visualization and manipulation of 3D objects represented by mesh or 3DGS within a unified framework, opening up new possibilities in embodied AI, vir tual reality, and gaming. We will release our source code to facilitate future research.
翻译:网格与3D高斯泼溅(3DGS)的联合渲染与变形具有重要价值,因为这两种表示方法在图形应用中具有互补优势。然而,由于表示形式和渲染管线的差异,现有研究分别渲染网格和3DGS,难以精确处理遮挡与透明效果。此外,变形的3DGS仍因对代理网格拓扑质量的敏感性而存在视觉伪影。这些问题严重阻碍了3DGS与网格的联合使用,使得3DGS难以适配传统的面向网格的图形管线。我们提出了UniMGS,这是首个以单通道抗锯齿方式栅格化网格与3DGS的统一框架,并基于代理网格提出了一种新颖的3DGS变形绑定策略。我们的核心思路是在单通道中通过抗锯齿α混合来融合三角形与高斯片段的颜色,从而获得视觉一致的结果,并精确处理遮挡与透明度。为改善变形后3DGS的视觉表现,我们以高斯为中心的绑定策略采用代理网格,并将高斯与网格面在空间上关联,显著减少了渲染伪影。通过这两个组件,UniMGS能够在统一框架内实现以网格或3DGS表示的3D物体的可视化与操控,为具身智能、虚拟现实和游戏领域开辟了新的可能性。我们将公开源代码以促进后续研究。