We present DIRECT-3D, a diffusion-based 3D generative model for creating high-quality 3D assets (represented by Neural Radiance Fields) from text prompts. Unlike recent 3D generative models that rely on clean and well-aligned 3D data, limiting them to single or few-class generation, our model is directly trained on extensive noisy and unaligned `in-the-wild' 3D assets, mitigating the key challenge (i.e., data scarcity) in large-scale 3D generation. In particular, DIRECT-3D is a tri-plane diffusion model that integrates two innovations: 1) A novel learning framework where noisy data are filtered and aligned automatically during the training process. Specifically, after an initial warm-up phase using a small set of clean data, an iterative optimization is introduced in the diffusion process to explicitly estimate the 3D pose of objects and select beneficial data based on conditional density. 2) An efficient 3D representation that is achieved by disentangling object geometry and color features with two separate conditional diffusion models that are optimized hierarchically. Given a prompt input, our model generates high-quality, high-resolution, realistic, and complex 3D objects with accurate geometric details in seconds. We achieve state-of-the-art performance in both single-class generation and text-to-3D generation. We also demonstrate that DIRECT-3D can serve as a useful 3D geometric prior of objects, for example to alleviate the well-known Janus problem in 2D-lifting methods such as DreamFusion. The code and models are available for research purposes at: https://github.com/qihao067/direct3d.

翻译：我们提出了DIRECT-3D，一个基于扩散的三维生成模型，用于从文本提示创建高质量的三维资产（以神经辐射场表示）。与近期依赖干净且对齐良好的三维数据、因而局限于单一或少数类别生成的三维生成模型不同，我们的模型直接在大量噪声且未对齐的“野外”三维资产上进行训练，从而缓解了大规模三维生成中的关键挑战（即数据稀缺问题）。具体而言，DIRECT-3D是一个三平面扩散模型，它整合了两项创新：1）一种新颖的学习框架，在训练过程中自动过滤和对齐噪声数据。具体来说，在使用少量干净数据进行初始预热阶段后，我们在扩散过程中引入了迭代优化，以显式估计物体的三维姿态，并基于条件密度选择有益的数据。2）一种高效的三维表示，通过使用两个独立的、分层优化的条件扩散模型来解耦物体的几何特征与颜色特征来实现。给定一个提示输入，我们的模型能在数秒内生成具有精确几何细节的高质量、高分辨率、逼真且复杂的三维物体。我们在单类别生成和文本到三维生成任务上均取得了最先进的性能。我们还证明了DIRECT-3D可以作为物体有用的三维几何先验，例如，用于缓解像DreamFusion这样的二维提升方法中众所周知的Janus问题。代码和模型已开源供研究使用，地址为：https://github.com/qihao067/direct3d。