In this paper, we address the problem of simultaneous relighting and novel view synthesis of a complex scene from multi-view images with a limited number of light sources. We propose an analysis-synthesis approach called Relit-NeuLF. Following the recent neural 4D light field network (NeuLF), Relit-NeuLF first leverages a two-plane light field representation to parameterize each ray in a 4D coordinate system, enabling efficient learning and inference. Then, we recover the spatially-varying bidirectional reflectance distribution function (SVBRDF) of a 3D scene in a self-supervised manner. A DecomposeNet learns to map each ray to its SVBRDF components: albedo, normal, and roughness. Based on the decomposed BRDF components and conditioning light directions, a RenderNet learns to synthesize the color of the ray. To self-supervise the SVBRDF decomposition, we encourage the predicted ray color to be close to the physically-based rendering result using the microfacet model. Comprehensive experiments demonstrate that the proposed method is efficient and effective on both synthetic data and real-world human face data, and outperforms the state-of-the-art results. We publicly released our code on GitHub. You can find it here: https://github.com/oppo-us-research/RelitNeuLF

翻译：本文针对从多视角图像（光源数量有限）中对复杂场景同时进行重光照与新视角合成的问题。我们提出了一种名为Relit-NeuLF的分析-合成方法。沿用近期提出的神经4D光场网络（NeuLF），Relit-NeuLF首先利用双平面光场表示在4D坐标系中对每条光线进行参数化，从而实现高效学习与推理。随后，我们以自监督方式恢复三维场景的空间变化双向反射分布函数（SVBRDF）。分解网络（DecomposeNet）学习将每条光线映射至其SVBRDF分量：反照率、法向及粗糙度。基于分解后的BRDF分量与条件化光照方向，渲染网络（RenderNet）学习合成光线颜色。为实现SVBRDF分解的自监督，我们使预测光线颜色逼近基于微面元模型的物理渲染结果。综合实验表明，该方法在合成数据及真实人脸数据上均高效且有效，并优于当前最优结果。我们已在GitHub上公开代码：https://github.com/oppo-us-research/RelitNeuLF