Face rendering using neural radiance fields (NeRF) is a rapidly developing research area in computer vision. While recent methods primarily focus on controlling facial attributes such as identity and expression, they often overlook the crucial aspect of modeling eyeball rotation, which holds importance for various downstream tasks. In this paper, we aim to learn a face NeRF model that is sensitive to eye movements from multi-view images. We address two key challenges in eye-aware face NeRF learning: how to effectively capture eyeball rotation for training and how to construct a manifold for representing eyeball rotation. To accomplish this, we first fit FLAME, a well-established parametric face model, to the multi-view images considering multi-view consistency. Subsequently, we introduce a new Dynamic Eye-aware NeRF (DeNeRF). DeNeRF transforms 3D points from different views into a canonical space to learn a unified face NeRF model. We design an eye deformation field for the transformation, including rigid transformation, e.g., eyeball rotation, and non-rigid transformation. Through experiments conducted on the ETH-XGaze dataset, we demonstrate that our model is capable of generating high-fidelity images with accurate eyeball rotation and non-rigid periocular deformation, even under novel viewing angles. Furthermore, we show that utilizing the rendered images can effectively enhance gaze estimation performance.

翻译：基于神经辐射场（NeRF）的人脸渲染是计算机视觉中快速发展的研究方向。现有方法主要聚焦于控制身份和表情等人脸属性，但往往忽略了眼球旋转建模这一关键方面，而这对各类下游任务具有重要意义。本文旨在从多视角图像中学习对人眼运动敏感的NeRF人脸模型。我们解决了眼部感知人脸NeRF学习中的两个关键挑战：如何有效捕捉眼球旋转以进行训练，以及如何构建表征眼球旋转的流形。为此，我们首先在考虑多视角一致性的前提下，将经典参数化人脸模型FLAME拟合至多视角图像，继而提出动态眼部感知NeRF（DeNeRF）。DeNeRF将来自不同视角的三维点转换至规范空间以学习统一的人脸NeRF模型，并针对该变换设计了包含刚体变换（如眼球旋转）与非刚体变换的眼部形变场。通过在ETH-XGaze数据集上的实验证明，即使在新的视角下，我们的模型仍能生成具有精确眼球旋转及非刚体眼周形变的高保真图像。此外，我们展示了利用渲染图像可有效提升视线估计性能。