Creating realistic 3D facial animation is crucial for various applications in the movie production and gaming industry, especially with the burgeoning demand in the metaverse. However, prevalent methods such as blendshape-based approaches and facial rigging techniques are time-consuming, labor-intensive, and lack standardized configurations, making facial animation production challenging and costly. In this paper, we propose a novel self-supervised framework, Versatile Face Animator, which combines facial motion capture with motion retargeting in an end-to-end manner, eliminating the need for blendshapes or rigs. Our method has the following two main characteristics: 1) we propose an RGBD animation module to learn facial motion from raw RGBD videos by hierarchical motion dictionaries and animate RGBD images rendered from 3D facial mesh coarse-to-fine, enabling facial animation on arbitrary 3D characters regardless of their topology, textures, blendshapes, and rigs; and 2) we introduce a mesh retarget module to utilize RGBD animation to create 3D facial animation by manipulating facial mesh with controller transformations, which are estimated from dense optical flow fields and blended together with geodesic-distance-based weights. Comprehensive experiments demonstrate the effectiveness of our proposed framework in generating impressive 3D facial animation results, highlighting its potential as a promising solution for the cost-effective and efficient production of facial animation in the metaverse.

翻译：创建逼真的3D面部动画对于电影制作和游戏行业中的各种应用至关重要，尤其是在元宇宙需求日益增长的背景下。然而，目前主流的方法（如基于混合形状的方法和面部绑定技术）耗时费力、缺乏标准化配置，导致面部动画制作既困难又成本高昂。在本文中，我们提出了一种新颖的自监督框架——多功能面部动画器，它以端到端的方式将面部动作捕捉与动作重定向相结合，无需使用混合形状或绑定。我们的方法具有以下两个主要特点：1）我们提出了一种RGBD动画模块，通过分层动作字典从原始RGBD视频中学习面部动作，并对从3D面部网格渲染的RGBD图像进行由粗到精的动画化，从而能够对任意3D角色进行面部动画制作，无论其拓扑结构、纹理、混合形状和绑定如何；2）我们引入了一个网格重定向模块，通过使用控制器变换来操作面部网格，从而利用RGBD动画创建3D面部动画，其中控制器变换是从密集光流场估计的，并基于测地距离权重进行混合。大量实验证明了我们提出的框架在生成令人印象深刻的3D面部动画结果方面的有效性，突显了其作为元宇宙中经济高效的面部动画制作解决方案的潜力。