Automatic 3D content creation has achieved rapid progress recently due to the availability of pre-trained, large language models and image diffusion models, forming the emerging topic of text-to-3D content creation. Existing text-to-3D methods commonly use implicit scene representations, which couple the geometry and appearance via volume rendering and are suboptimal in terms of recovering finer geometries and achieving photorealistic rendering; consequently, they are less effective for generating high-quality 3D assets. In this work, we propose a new method of Fantasia3D for high-quality text-to-3D content creation. Key to Fantasia3D is the disentangled modeling and learning of geometry and appearance. For geometry learning, we rely on a hybrid scene representation, and propose to encode surface normal extracted from the representation as the input of the image diffusion model. For appearance modeling, we introduce the spatially varying bidirectional reflectance distribution function (BRDF) into the text-to-3D task, and learn the surface material for photorealistic rendering of the generated surface. Our disentangled framework is more compatible with popular graphics engines, supporting relighting, editing, and physical simulation of the generated 3D assets. We conduct thorough experiments that show the advantages of our method over existing ones under different text-to-3D task settings. Project page and source codes: https://fantasia3d.github.io/.

翻译：自动3D内容创建近日因预训练大语言模型和图像扩散模型的发展取得了快速进展，形成了新兴的文本到3D内容生成研究方向。现有文本到3D方法通常采用隐式场景表示，通过体渲染耦合几何与外观，在恢复精细几何和实现逼真渲染方面表现欠佳，因此难以高效生成高质量3D资产。本文提出了一种名为Fantasia3D的新方法，用于实现高质量文本到3D内容生成。Fantasia3D的核心在于对几何与外观进行解耦建模与学习。在几何学习方面，我们采用混合场景表示，并提出将从中提取的表面法线编码作为图像扩散模型的输入。在外观建模方面，我们将空间变化双向反射分布函数（BRDF）引入文本到3D任务，并学习表面材质以实现生成表面的逼真渲染。我们的解耦框架与主流图形引擎更兼容，支持对生成3D资产进行重光照、编辑和物理仿真。我们通过全面实验展示了本方法在不同文本到3D任务设置下相较于现有方法的优势。项目页面与源代码：https://fantasia3d.github.io/。