We propose GauFace, a novel Gaussian Splatting representation, tailored for efficient animation and rendering of physically-based facial assets. Leveraging strong geometric priors and constrained optimization, GauFace ensures a neat and structured Gaussian representation, delivering high fidelity and real-time facial interaction of 30fps@1440p on a Snapdragon 8 Gen 2 mobile platform. Then, we introduce TransGS, a diffusion transformer that instantly translates physically-based facial assets into the corresponding GauFace representations. Specifically, we adopt a patch-based pipeline to handle the vast number of Gaussians effectively. We also introduce a novel pixel-aligned sampling scheme with UV positional encoding to ensure the throughput and rendering quality of GauFace assets generated by our TransGS. Once trained, TransGS can instantly translate facial assets with lighting conditions to GauFace representation, With the rich conditioning modalities, it also enables editing and animation capabilities reminiscent of traditional CG pipelines. We conduct extensive evaluations and user studies, compared to traditional offline and online renderers, as well as recent neural rendering methods, which demonstrate the superior performance of our approach for facial asset rendering. We also showcase diverse immersive applications of facial assets using our TransGS approach and GauFace representation, across various platforms like PCs, phones and even VR headsets.

翻译：我们提出GauFace，一种新颖的高斯泼溅表示方法，专为基于物理的面部资产的高效动画与渲染而设计。通过利用强几何先验与约束优化，GauFace确保了整洁且结构化的高斯表示，在骁龙8 Gen 2移动平台上实现了1440p分辨率下30帧/秒的高保真实时面部交互。随后，我们引入TransGS，一种扩散Transformer模型，能够即时将基于物理的面部资产转换为对应的GauFace表示。具体而言，我们采用基于图像块的流水线以有效处理海量高斯元素，并提出了结合UV位置编码的新型像素对齐采样方案，确保TransGS生成的GauFace资产在吞吐量与渲染质量上的表现。经训练后，TransGS可即时将带光照条件的面部资产转换为GauFace表示，其丰富的条件模态还支持类似传统CG流程的编辑与动画功能。通过与传统离线/在线渲染器及近期神经渲染方法的广泛对比评估与用户研究，验证了本方法在面部资产渲染方面的卓越性能。我们还在PC、手机乃至VR头显等多平台上展示了基于TransGS方法与GauFace表示的面部资产在多样化沉浸式应用中的潜力。