Creating relightable urban scenes from images or videos is widely useful but highly ill-posed. Urban environments are typically unbounded and extend beyond the visible regions. As a result, many portions of the scene remain unobserved, yet these invisible regions can cast shadows onto visible areas. Reasonably modeling shadows cast by such invisible regions is challenging and poses a significant obstacle to creating relightable urban scenes. At the same time, sparse input views and complex illumination conditions further complicate relighting, as they introduce severe ambiguities in material decomposition. In this paper, we propose Shadow-guided Relightable Urban Scene with Generation model (SRUG), a novel framework designed to address relighting challenges in urban scenes. SRUG leverages shadows to guide a 3D completion model for recovering the geometry of invisible regions, promoting the synthesis of physically reasonable shadows. In addition, SRUG employs an iterative material decomposition scheme that applies the large material model (LMM) to provide material supervision and iteratively decompose the scene's material properties, enabling robust material decomposition. Building upon these components, we introduce a physically-based lighting model that captures the complex illumination of urban scenes and supports reliable relighting. Extensive quantitative evaluations and visual comparisons demonstrate that our method outperforms existing approaches in both novel view synthesis and relighting tasks.

翻译：从图像或视频中创建可重光照的城市场景具有广泛实用性，但存在严重的病态性。城市环境通常是无界且超出可见区域的，导致场景中大量部分未被观测到，而这些不可见区域可能向可见区域投射阴影。合理建模这类由不可见区域投射的阴影极具挑战性，成为构建可重光照城市场景的主要障碍。同时，稀疏输入视角与复杂光照条件进一步加剧了重光照难度，因其在材质分解中引入严重歧义。本文提出基于阴影引导的生成模型可重光照城市场景框架（SRUG），该创新框架旨在解决城市场景重光照挑战。SRUG利用阴影引导三维补全模型恢复不可见区域的几何结构，促进符合物理规律的阴影合成。此外，SRUG采用迭代材质分解方案，通过大材质模型（LMM）提供材质监督，逐步分解场景材质属性，实现稳健的材质分解。基于上述组件，我们引入基于物理的光照模型，能够捕捉城市场景的复杂光照并支持可靠的重光照。大量定量评估与视觉比较表明，本方法在新视角合成与重光照任务上均优于现有方法。