Creating relightable urban scenes from images or videos is widely useful but highly ill-posed. Urban environments are typically unbounded and extend beyond the visible regions. As a result, many portions of the scene remain unobserved, yet these invisible regions can cast shadows onto visible areas. Reasonably modeling shadows cast by such invisible regions is challenging and poses a significant obstacle to creating relightable urban scenes. At the same time, sparse input views and complex illumination conditions further complicate relighting, as they introduce severe ambiguities in material decomposition. In this paper, we propose Shadow-guided Relightable Urban Scene with Generation model (SRUG), a novel framework designed to address relighting challenges in urban scenes. SRUG leverages shadows to guide a 3D completion model for recovering the geometry of invisible regions, promoting the synthesis of physically reasonable shadows. In addition, SRUG employs an iterative material decomposition scheme that applies the large material model (LMM) to provide material supervision and iteratively decompose the scene's material properties, enabling robust material decomposition. Building upon these components, we introduce a physically-based lighting model that captures the complex illumination of urban scenes and supports reliable relighting. Extensive quantitative evaluations and visual comparisons demonstrate that our method outperforms existing approaches in both novel view synthesis and relighting tasks.

翻译：从图像或视频中创建可重光照的城市场景具有广泛应用价值，但高度不适定。城市环境通常是无界且超出可见区域的。因此，场景的许多部分未被观测到，但这些不可见区域可能向可见区域投射阴影。合理模拟这些不可见区域投射的阴影颇具挑战性，并成为创建可重光照城市场景的重大障碍。同时，稀疏的输入视图和复杂的照明条件进一步加剧了重光照的难度，因为它们会在材质分解中引入严重歧义。本文提出阴影引导的可重光照城市场景生成模型（SRUG），一种旨在解决城市场景重光照挑战的新型框架。SRUG利用阴影引导三维补全模型恢复不可见区域的几何结构，促进物理合理阴影的合成。此外，SRUG采用迭代材质分解方案，应用大型材质模型（LMM）提供材质监督并迭代分解场景材质属性，从而实现鲁棒的材质分解。基于这些组件，我们引入了一种基于物理的照明模型，该模型可捕捉城市场景的复杂照明并支持可靠的重光照。广泛的定量评估和视觉比较表明，我们的方法在新视角合成和重光照任务中均优于现有方法。