Diffusion models have emerged as the best approach for generative modeling of 2D images. Part of their success is due to the possibility of training them on millions if not billions of images with a stable learning objective. However, extending these models to 3D remains difficult for two reasons. First, finding a large quantity of 3D training data is much more complex than for 2D images. Second, while it is conceptually trivial to extend the models to operate on 3D rather than 2D grids, the associated cubic growth in memory and compute complexity makes this infeasible. We address the first challenge by introducing a new diffusion setup that can be trained, end-to-end, with only posed 2D images for supervision; and the second challenge by proposing an image formation model that decouples model memory from spatial memory. We evaluate our method on real-world data, using the CO3D dataset which has not been used to train 3D generative models before. We show that our diffusion models are scalable, train robustly, and are competitive in terms of sample quality and fidelity to existing approaches for 3D generative modeling.

翻译：扩散模型已成为二维图像生成建模的最佳方法。其成功部分归功于能够在稳定学习目标下对数百万甚至数十亿张图像进行训练。然而，将这些模型扩展到三维仍面临两大困难。首先，获取大量三维训练数据比二维图像复杂得多。其次，虽然从概念上而言将模型从二维网格扩展至三维网格是直接的，但随之而来的内存和计算复杂度呈立方增长使得这一方法不可行。针对第一个挑战，我们提出了一种新的扩散设置，该设置仅需带位姿的二维图像作为监督即可实现端到端训练；针对第二个挑战，我们提出了一种图像形成模型，该模型将模型内存与空间内存解耦。我们使用此前未用于训练三维生成模型的CO3D数据集，在我们的方法上进行了真实世界数据评估。实验表明，我们的扩散模型具备可扩展性、训练鲁棒性，并在样本质量和保真度方面与现有的三维生成建模方法具有竞争力。