We address the problem of learning person-specific facial priors from a small number (e.g., 20) of portrait photos of the same person. This enables us to edit this specific person's facial appearance, such as expression and lighting, while preserving their identity and high-frequency facial details. Key to our approach, which we dub DiffusionRig, is a diffusion model conditioned on, or "rigged by," crude 3D face models estimated from single in-the-wild images by an off-the-shelf estimator. On a high level, DiffusionRig learns to map simplistic renderings of 3D face models to realistic photos of a given person. Specifically, DiffusionRig is trained in two stages: It first learns generic facial priors from a large-scale face dataset and then person-specific priors from a small portrait photo collection of the person of interest. By learning the CGI-to-photo mapping with such personalized priors, DiffusionRig can "rig" the lighting, facial expression, head pose, etc. of a portrait photo, conditioned only on coarse 3D models while preserving this person's identity and other high-frequency characteristics. Qualitative and quantitative experiments show that DiffusionRig outperforms existing approaches in both identity preservation and photorealism. Please see the project website: https://diffusionrig.github.io for the supplemental material, video, code, and data.

翻译：我们研究了从同一人物的少量（例如20张）肖像照片中学习个体特异性面部先验的问题。这使得我们能够编辑该特定人物的面部外观（如表情和光照），同时保留其身份特征与高频面部细节。我们提出的方法名为DiffusionRig，其核心是一个扩散模型，该模型以现成方法从单张野外图像中估计的粗略3D人脸模型为条件（即"被绑定"）。从宏观来看，DiffusionRig学习将3D人脸模型的简略渲染结果映射到给定人物的真实照片。具体而言，DiffusionRig分两阶段训练：首先从大规模人脸数据集中学习通用面部先验，随后从目标人物的小规模肖像照片集中学习个体特异性先验。通过利用这种个性化先验学习CGI到照片的映射，DiffusionRig能够仅以粗略3D模型为条件，"绑定"肖像照片的光照、面部表情、头部姿态等属性，同时保留该人物的身份特征及其他高频细节。定性与定量实验表明，DiffusionRig在身份保留与照片逼真度方面均优于现有方法。补充材料、视频、代码及数据详见项目网站：https://diffusionrig.github.io。