Generating photorealistic images with controllable camera pose and scene contents is essential for many applications including AR/VR and simulation. Despite the fact that rapid progress has been made in 3D-aware generative models, most existing methods focus on object-centric images and are not applicable to generating urban scenes for free camera viewpoint control and scene editing. To address this challenging task, we propose UrbanGIRAFFE, which uses a coarse 3D panoptic prior, including the layout distribution of uncountable stuff and countable objects, to guide a 3D-aware generative model. Our model is compositional and controllable as it breaks down the scene into stuff, objects, and sky. Using stuff prior in the form of semantic voxel grids, we build a conditioned stuff generator that effectively incorporates the coarse semantic and geometry information. The object layout prior further allows us to learn an object generator from cluttered scenes. With proper loss functions, our approach facilitates photorealistic 3D-aware image synthesis with diverse controllability, including large camera movement, stuff editing, and object manipulation. We validate the effectiveness of our model on both synthetic and real-world datasets, including the challenging KITTI-360 dataset.

翻译：生成具有可控相机姿态和场景内容的光真实感图像对于增强现实/虚拟现实及仿真等众多应用至关重要。尽管3D感知生成模型已取得快速进展，但现有方法大多聚焦于以物体为中心的图像，无法生成支持自由相机视角控制和场景编辑的城市场景。为应对这一挑战，我们提出UrbanGIRAFFE，利用包含不可数物质与可数物体布局分布的粗粒度3D全景先验来引导3D感知生成模型。该模型具有组合性与可控性，将场景分解为物质、物体和天空。通过采用语义体素网格形式的物质先验，我们构建了条件式物质生成器，有效融合了粗粒度语义与几何信息。物体布局先验使我们能够从杂乱场景中学习物体生成器。借助适当的损失函数，我们的方法实现了具有多样可控性（包括大范围相机移动、物质编辑与物体操控）的光真实感3D感知图像合成。我们在合成与真实世界数据集（包括具有挑战性的KITTI-360数据集）上验证了模型的有效性。