This paper presents ER-NeRF, a novel conditional Neural Radiance Fields (NeRF) based architecture for talking portrait synthesis that can concurrently achieve fast convergence, real-time rendering, and state-of-the-art performance with small model size. Our idea is to explicitly exploit the unequal contribution of spatial regions to guide talking portrait modeling. Specifically, to improve the accuracy of dynamic head reconstruction, a compact and expressive NeRF-based Tri-Plane Hash Representation is introduced by pruning empty spatial regions with three planar hash encoders. For speech audio, we propose a Region Attention Module to generate region-aware condition feature via an attention mechanism. Different from existing methods that utilize an MLP-based encoder to learn the cross-modal relation implicitly, the attention mechanism builds an explicit connection between audio features and spatial regions to capture the priors of local motions. Moreover, a direct and fast Adaptive Pose Encoding is introduced to optimize the head-torso separation problem by mapping the complex transformation of the head pose into spatial coordinates. Extensive experiments demonstrate that our method renders better high-fidelity and audio-lips synchronized talking portrait videos, with realistic details and high efficiency compared to previous methods.

翻译：本文提出ER-NeRF，一种基于条件神经辐射场（NeRF）的新型说话人肖像合成架构，能够同时实现快速收敛、实时渲染以及小模型尺寸下的最先进性能。我们的核心思想是显式利用空间区域的不等贡献来指导说话人肖像建模。具体而言，为提升动态头部重建精度，我们引入了一种紧凑且表达力强的NeRF三平面哈希表示，通过三个平面哈希编码器剪除空域空间区域。针对语音音频，我们提出区域注意力模块，通过注意力机制生成区域感知条件特征。不同于现有方法使用基于MLP的编码器隐式学习跨模态关系，注意力机制在音频特征与空间区域之间建立显式连接，以捕获局部运动的先验知识。此外，我们引入一种直接且快速的适应性姿态编码，通过将头部姿态的复杂变换映射到空间坐标，优化头-躯干分离问题。大量实验表明，与先前方法相比，我们的方法能够渲染出更逼真且音唇同步的说话人肖像视频，同时具备逼真细节和高效率。