In this work, we present SceneDreamer, an unconditional generative model for unbounded 3D scenes, which synthesizes large-scale 3D landscapes from random noise. Our framework is learned from in-the-wild 2D image collections only, without any 3D annotations. At the core of SceneDreamer is a principled learning paradigm comprising 1) an efficient yet expressive 3D scene representation, 2) a generative scene parameterization, and 3) an effective renderer that can leverage the knowledge from 2D images. Our approach begins with an efficient bird's-eye-view (BEV) representation generated from simplex noise, which includes a height field for surface elevation and a semantic field for detailed scene semantics. This BEV scene representation enables 1) representing a 3D scene with quadratic complexity, 2) disentangled geometry and semantics, and 3) efficient training. Moreover, we propose a novel generative neural hash grid to parameterize the latent space based on 3D positions and scene semantics, aiming to encode generalizable features across various scenes. Lastly, a neural volumetric renderer, learned from 2D image collections through adversarial training, is employed to produce photorealistic images. Extensive experiments demonstrate the effectiveness of SceneDreamer and superiority over state-of-the-art methods in generating vivid yet diverse unbounded 3D worlds.

翻译：本文提出SceneDreamer，一种用于无界三维场景的无条件生成模型，能够从随机噪声合成大规模三维景观。该框架仅从野外二维图像集合中学习，无需任何三维标注。SceneDreamer的核心是一个原则性学习范式，包含：1）高效且富有表现力的三维场景表征，2）生成式场景参数化方法，3）可充分利用二维图像知识的有效渲染器。我们的方法首先利用单纯形噪声生成高效的鸟瞰图（BEV）表征，该表征包含用于地表高程的高度场和用于精细场景语义的语义场。这种BEV场景表征能够实现：1）以二次复杂度表示三维场景，2）解耦几何与语义，3）高效训练。此外，我们提出一种新颖的生成式神经哈希网格，基于三维位置和场景语义对隐空间进行参数化，旨在编码跨场景的通用特征。最后，通过对抗训练从二维图像集合中学习神经体积渲染器，用于生成逼真图像。大量实验证明了SceneDreamer的有效性，以及其在生成生动多样的无界三维世界方面相较于最先进方法的优越性。