At the core of portrait photography is the search for ideal lighting and viewpoint. The process often requires advanced knowledge in photography and an elaborate studio setup. In this work, we propose Holo-Relighting, a volumetric relighting method that is capable of synthesizing novel viewpoints, and novel lighting from a single image. Holo-Relighting leverages the pretrained 3D GAN (EG3D) to reconstruct geometry and appearance from an input portrait as a set of 3D-aware features. We design a relighting module conditioned on a given lighting to process these features, and predict a relit 3D representation in the form of a tri-plane, which can render to an arbitrary viewpoint through volume rendering. Besides viewpoint and lighting control, Holo-Relighting also takes the head pose as a condition to enable head-pose-dependent lighting effects. With these novel designs, Holo-Relighting can generate complex non-Lambertian lighting effects (e.g., specular highlights and cast shadows) without using any explicit physical lighting priors. We train Holo-Relighting with data captured with a light stage, and propose two data-rendering techniques to improve the data quality for training the volumetric relighting system. Through quantitative and qualitative experiments, we demonstrate Holo-Relighting can achieve state-of-the-arts relighting quality with better photorealism, 3D consistency and controllability.

翻译：人像摄影的核心在于寻找理想的光照与视角，这一过程通常需要专业的摄影知识和复杂的影棚设备。本文提出Holo-Relighting，一种能够从单张图像合成新视角与新光照的体积重照明方法。该方法利用预训练的3D GAN（EG3D）从输入人像中重建几何与外观，提取一组三维感知特征。我们设计了一个以给定光照为条件的重照明模块，对这些特征进行处理，并预测以三平面形式表征的重照明三维表示，通过体渲染可渲染至任意视角。除视角与光照控制外，Holo-Relighting还以头部姿态为条件，实现依赖于头部姿态的光照效果。通过这些创新设计，Holo-Relighting无需显式物理光照先验即可生成复杂的非朗伯光照效果（如镜面高光和投射阴影）。我们使用光场采集数据训练Holo-Relighting，并提出两种数据渲染技术以提升训练体积重照明系统的数据质量。通过定量与定性实验证明，Holo-Relighting能够达到最优的重照明质量，具备更逼真的光影效果、三维一致性与可控性。