The creation of high-fidelity, digital versions of human heads is an important stepping stone in the process of further integrating virtual components into our everyday lives. Constructing such avatars is a challenging research problem, due to a high demand for photo-realism and real-time rendering performance. In this work, we propose Neural Parametric Gaussian Avatars (NPGA), a data-driven approach to create high-fidelity, controllable avatars from multi-view video recordings. We build our method around 3D Gaussian Splatting for its highly efficient rendering and to inherit the topological flexibility of point clouds. In contrast to previous work, we condition our avatars' dynamics on the rich expression space of neural parametric head models (NPHM), instead of mesh-based 3DMMs. To this end, we distill the backward deformation field of our underlying NPHM into forward deformations which are compatible with rasterization-based rendering. All remaining fine-scale, expression-dependent details are learned from the multi-view videos. To increase the representational capacity of our avatars, we augment the canonical Gaussian point cloud using per-primitive latent features which govern its dynamic behavior. To regularize this increased dynamic expressivity, we propose Laplacian terms on the latent features and predicted dynamics. We evaluate our method on the public NeRSemble dataset, demonstrating that NPGA significantly outperforms the previous state-of-the-art avatars on the self-reenactment task by 2.6 PSNR. Furthermore, we demonstrate accurate animation capabilities from real-world monocular videos.

翻译：创建高保真度的人头数字模型是将虚拟组件进一步融入日常生活的关键步骤。由于对照片级真实感和实时渲染性能的高要求，构建此类化身是一个具有挑战性的研究问题。在本工作中，我们提出了神经参数化高斯化身（NPGA），一种从多视角视频记录中创建高保真度、可控化身的数驱方法。我们基于3D高斯泼溅构建方法，以利用其高效渲染能力并继承点云的拓扑灵活性。与先前工作不同，我们将化身动态条件建立在神经参数化头部模型（NPHM）的丰富表情空间上，而非基于网格的3D形变模型。为此，我们将底层NPHM的后向形变场提炼为与基于光栅化的渲染相兼容的前向形变。所有剩余的细粒度、表情相关的细节均从多视角视频中学习。为提升化身的表征能力，我们使用控制其动态行为的逐基元潜在特征来增强规范高斯点云。为规范这种增强的动态表达能力，我们提出了针对潜在特征和预测动态的拉普拉斯正则项。我们在公开的NeRSemble数据集上评估了所提方法，结果表明NPGA在自我重演任务上以2.6 PSNR显著优于先前最先进的化身方法。此外，我们展示了从真实世界单目视频中实现精确动画生成的能力。