3D Content Generation is at the heart of many computer graphics applications, including video gaming, film-making, virtual and augmented reality, etc. This paper proposes a novel deep-learning based approach for automatically generating interactive and playable 3D game scenes, all from the user's casual prompts such as a hand-drawn sketch. Sketch-based input offers a natural, and convenient way to convey the user's design intention in the content creation process. To circumvent the data-deficient challenge in learning (i.e. the lack of large training data of 3D scenes), our method leverages a pre-trained 2D denoising diffusion model to generate a 2D image of the scene as the conceptual guidance. In this process, we adopt the isometric projection mode to factor out unknown camera poses while obtaining the scene layout. From the generated isometric image, we use a pre-trained image understanding method to segment the image into meaningful parts, such as off-ground objects, trees, and buildings, and extract the 2D scene layout. These segments and layouts are subsequently fed into a procedural content generation (PCG) engine, such as a 3D video game engine like Unity or Unreal, to create the 3D scene. The resulting 3D scene can be seamlessly integrated into a game development environment and is readily playable. Extensive tests demonstrate that our method can efficiently generate high-quality and interactive 3D game scenes with layouts that closely follow the user's intention.

翻译：三维内容生成是视频游戏、电影制作、虚拟与增强现实等众多计算机图形应用的核心。本文提出一种基于深度学习的新方法，能够仅根据用户提供的随意提示（例如手绘草图）自动生成可交互、可游玩的3D游戏场景。草图输入为内容创作过程中传达用户设计意图提供了一种自然且便捷的方式。为应对学习过程中数据匮乏的挑战（即缺乏大规模3D场景训练数据），本方法利用预训练的二维去噪扩散模型生成场景的二维图像作为概念指导。在此过程中，我们采用等轴投影模式以消除未知相机姿态的影响，同时获取场景布局。基于生成的等轴图像，我们使用预训练的图像理解方法将图像分割为有意义的组成部分（如离地物体、树木、建筑物等），并提取二维场景布局。这些分割结果与布局信息随后被输入程序化内容生成引擎（例如Unity或Unreal等3D视频游戏引擎）以创建三维场景。生成的3D场景可无缝集成至游戏开发环境，并具备即时的可玩性。大量测试表明，本方法能够高效生成高质量、可交互的3D游戏场景，其布局紧密贴合用户的设计意图。