We present a novel approach to single-view face relighting in the wild. Handling non-diffuse effects, such as global illumination or cast shadows, has long been a challenge in face relighting. Prior work often assumes Lambertian surfaces, simplified lighting models or involves estimating 3D shape, albedo, or a shadow map. This estimation, however, is error-prone and requires many training examples with lighting ground truth to generalize well. Our work bypasses the need for accurate estimation of intrinsic components and can be trained solely on 2D images without any light stage data, multi-view images, or lighting ground truth. Our key idea is to leverage a conditional diffusion implicit model (DDIM) for decoding a disentangled light encoding along with other encodings related to 3D shape and facial identity inferred from off-the-shelf estimators. We also propose a novel conditioning technique that eases the modeling of the complex interaction between light and geometry by using a rendered shading reference to spatially modulate the DDIM. We achieve state-of-the-art performance on standard benchmark Multi-PIE and can photorealistically relight in-the-wild images. Please visit our page: https://diffusion-face-relighting.github.io

翻译：我们提出一种针对野外单视图面部重光照的新颖方法。处理非漫射效应（如全局光照或投影阴影）长期以来一直是面部重光照领域的挑战。先前工作通常假设朗伯表面、简化光照模型，或涉及三维形状、反照率及阴影图的估计。然而，此类估计易产生误差，且需要大量带有光照真值的训练样本才能良好泛化。我们的工作绕过了对内在分量精确估计的需求，仅需二维图像即可完成训练，无需任何光照舞台数据、多视图图像或光照真值。核心思想是利用条件扩散隐式模型（DDIM）对解耦的光照编码，以及由现成估计器推断的三维形状和面部身份相关编码进行解码。我们还提出一种新颖的条件化技术，通过使用可渲染着色参考图对DDIM进行空间调制，从而简化光照与几何之间复杂交互的建模过程。我们在标准基准数据集Multi-PIE上取得了最先进性能，并能对野外真实图像实现照片级真实感的重新光照。请访问我们的页面：https://diffusion-face-relighting.github.io