Virtual production (VP) use LED walls to provide both background imagery and image-based lighting. While this enables on-set compositing, it couples lighting to background and scene appearance, limiting flexibility for downstream editing. In addition, inverse rendering conventionally relies on physically-based rendering to estimates 3D geometry and lighting, using environment maps. However, these maps are typically low-resolution and assume far-field lighting. In VP, with near-field and high-resolution image-based lighting, this can lead to inaccuracies and introduce complexities when editing. Addressing this, we propose a VP-specific framework for 3D reconstruction and relighting using Gaussian Splatting. This uses the known background imagery to condition the relighting process. This avoids relying on environment maps and reduces compositing to a background-image editing task. To realize our framework, we introduce a process (and associated dataset) that captures real VP scenes under varying background content and illumination conditions. This data is used to decompose a 3D scene into fixed appearance and variable lighting components. The variable lighting process simulates light transport by parameterizing each primitive with a UV coordinate, intensity value and resolution modifier. Using mipmaps, these directly sample the background texture in image space - implicitly capturing reflections and refractions without physically-based rendering. Combined with the fixed appearance component, this allows us to render relit scenes using a Gaussian Splatting rasterizer. Compared to baselines, our approach achieves higher-quality 3D reconstruction and controllable relighting. The method is efficient (<3 GB RAM, <5 GB VRAM, <2 hours training, ~35 FPS) and supports rendering useful arbitrary output variables including depth, lighting intensity, lighting color, and unlit renders.

翻译：虚拟制作（VP）利用LED幕墙同时提供背景影像与基于图像的光照。这种方法虽能实现现场合成，却将光照与背景和场景外观耦合，限制了下游编辑的灵活性。此外，传统逆向渲染通常依赖基于物理的渲染技术，通过环境贴图估算三维几何与光照。然而，这些贴图通常分辨率较低，并假定为远场照明。在采用近场高分辨率图像照明的虚拟制作场景中，这种方法会导致精度损失并增加编辑复杂度。针对这一问题，我们提出一种面向虚拟制作的特定框架，基于高斯散射实现三维重建与重光照。该框架利用已知背景影像约束重光照过程，避免依赖环境贴图，将合成任务简化为背景影像编辑。为实现该框架，我们引入一套流程（及配套数据集），在可变背景内容与光照条件下采集真实虚拟制作场景数据。这些数据用于将三维场景分解为固定外观分量与可变光照分量。其中，可变光照过程通过为每个图元参数化UV坐标、强度值与分辨率修正因子来模拟光传输。借助mipmap技术，该方法在图像空间中直接采样背景纹理——无需基于物理的渲染即可隐式捕获反射与折射。与固定外观分量结合后，可通过高斯散射光栅化器渲染重光照场景。与基准方法相比，本方法实现了更高质量的三维重建与可控重光照。该方案效率优异（内存<3GB、显存<5GB、训练<2小时、帧率~35FPS），并支持渲染深度、光照强度、光照颜色及无光照渲染等任意实用变量。