Recent progress in 3D scene understanding enables scalable learning of representations across large datasets of diverse scenes. As a consequence, generalization to unseen scenes and objects, rendering novel views from just a single or a handful of input images, and controllable scene generation that supports editing, is now possible. However, training jointly on a large number of scenes typically compromises rendering quality when compared to single-scene optimized models such as NeRFs. In this paper, we leverage recent progress in diffusion models to equip 3D scene representation learning models with the ability to render high-fidelity novel views, while retaining benefits such as object-level scene editing to a large degree. In particular, we propose DORSal, which adapts a video diffusion architecture for 3D scene generation conditioned on frozen object-centric slot-based representations of scenes. On both complex synthetic multi-object scenes and on the real-world large-scale Street View dataset, we show that DORSal enables scalable neural rendering of 3D scenes with object-level editing and improves upon existing approaches.

翻译：近期三维场景理解领域的进展使得在多样化场景的大规模数据集中可扩展地学习表示成为可能。由此，对未见场景和物体的泛化、仅凭单张或少量输入图像生成新视角的渲染，以及支持编辑的可控场景生成已成为现实。然而，与NeRF等单场景优化模型相比，对大量场景的联合训练通常会牺牲渲染质量。本文利用扩散模型的最新进展，赋予三维场景表示学习模型高保真新视角渲染能力，同时在很大程度上保留物体级场景编辑等优势。具体而言，我们提出DORSal，该方法采用视频扩散架构进行三维场景生成，并以冻结的基于对象的场景槽表示作为条件。在复杂的合成多对象场景以及真实世界的大规模街景数据集上，实验表明DORSal能以物体级编辑能力实现可扩展的三维场景神经渲染，并优于现有方法。