The fidelity of relighting is bounded by both geometry and appearance representations. For geometry, both mesh and volumetric approaches have difficulty modeling intricate structures like 3D hair geometry. For appearance, existing relighting models are limited in fidelity and often too slow to render in real-time with high-resolution continuous environments. In this work, we present Relightable Gaussian Codec Avatars, a method to build high-fidelity relightable head avatars that can be animated to generate novel expressions. Our geometry model based on 3D Gaussians can capture 3D-consistent sub-millimeter details such as hair strands and pores on dynamic face sequences. To support diverse materials of human heads such as the eyes, skin, and hair in a unified manner, we present a novel relightable appearance model based on learnable radiance transfer. Together with global illumination-aware spherical harmonics for the diffuse components, we achieve real-time relighting with spatially all-frequency reflections using spherical Gaussians. This appearance model can be efficiently relit under both point light and continuous illumination. We further improve the fidelity of eye reflections and enable explicit gaze control by introducing relightable explicit eye models. Our method outperforms existing approaches without compromising real-time performance. We also demonstrate real-time relighting of avatars on a tethered consumer VR headset, showcasing the efficiency and fidelity of our avatars.

翻译：重光照的逼真度受限于几何与外观表征。在几何方面，网格和体积方法都难以建模如3D头发几何等复杂结构。在外观方面，现有重光照模型在逼真度上存在局限，且通常难以在实时高分辨率连续环境中高效渲染。本文提出可重光照高斯基编码虚拟人（Relightable Gaussian Codec Avatars），该方法能够构建高逼真度、可动画生成新表情的可重光照头部虚拟人。基于3D高斯的几何模型可捕获动态面部序列中具有3D一致性的亚毫米级细节，如发丝和毛孔。为统一处理人眼、皮肤、毛发等不同材质，我们提出基于可学习辐射度传输的新型可重光照外观模型。结合针对漫反射分量的全局光照感知球谐函数，我们利用球面高斯实现包含空间全频反射的实时重光照。该外观模型可在点光源和连续光照下高效重光照。通过引入可重光照显式眼球模型，我们进一步提升了眼部反射的逼真度并实现显式视线控制。在不牺牲实时性能的前提下，本方法优于现有技术。我们还展示了在连接消费级VR头显设备上的实时虚拟人重光照效果，验证了本方法的高效性与高保真度。