Creating relightable urban scenes from images or videos is widely useful but highly ill-posed. Urban environments are typically unbounded and extend beyond the visible regions. As a result, many portions of the scene remain unobserved, yet these invisible regions can cast shadows onto visible areas. Reasonably modeling shadows cast by such invisible regions is challenging and poses a significant obstacle to creating relightable urban scenes. At the same time, sparse input views and complex illumination conditions further complicate relighting, as they introduce severe ambiguities in material decomposition. In this paper, we propose Shadow-guided Relightable Urban Scene with Generation model (SRUG), a novel framework designed to address relighting challenges in urban scenes. SRUG leverages shadows to guide a 3D completion model for recovering the geometry of invisible regions, promoting the synthesis of physically reasonable shadows. In addition, SRUG employs an iterative material decomposition scheme that applies the large material model (LMM) to provide material supervision and iteratively decompose the scene's material properties, enabling robust material decomposition. Building upon these components, we introduce a physically-based lighting model that captures the complex illumination of urban scenes and supports reliable relighting. Extensive quantitative evaluations and visual comparisons demonstrate that our method outperforms existing approaches in both novel view synthesis and relighting tasks.

翻译：从图像或视频中创建可重光照的城市场景具有广泛用途，但属于高度不适定问题。城市环境通常是无边界的，范围远超可见区域，导致场景中大量部分未被观测到，而这些不可见区域却可能向可见区域投射阴影。合理建模这类不可见区域投射的阴影极具挑战性，并构成了创建可重光照城市场景的重大障碍。同时，稀疏输入视图与复杂光照条件进一步增加了重光照的难度，因为它们会为材质分解引入严重的歧义性。本文提出"基于阴影引导与生成模型的可重光照城市场景"框架（SRUG），该系统专门设计用于解决城市场景中的重光照挑战。SRUG利用阴影引导三维补全模型恢复不可见区域的几何结构，从而促进生成物理合理的阴影。此外，SRUG采用迭代材质分解方案，通过引入大规模材质模型（LMM）提供材质监督，并迭代分解场景的材质属性，实现了鲁棒的材质分解。基于这些组件，我们提出了基于物理的光照模型，该模型能够捕捉城市场景的复杂光照条件并支持可靠的重光照。大量定量评估与视觉对比表明，本方法在新视角合成与重光照任务中均优于现有方法。