Creating realistic 3D objects and clothed avatars from a single RGB image is an attractive yet challenging problem. Due to its ill-posed nature, recent works leverage powerful prior from 2D diffusion models pretrained on large datasets. Although 2D diffusion models demonstrate strong generalization capability, they cannot guarantee the generated multi-view images are 3D consistent. In this paper, we propose Gen-3Diffusion: Realistic Image-to-3D Generation via 2D & 3D Diffusion Synergy. We leverage a pre-trained 2D diffusion model and a 3D diffusion model via our elegantly designed process that synchronizes two diffusion models at both training and sampling time. The synergy between the 2D and 3D diffusion models brings two major advantages: 1) 2D helps 3D in generalization: the pretrained 2D model has strong generalization ability to unseen images, providing strong shape priors for the 3D diffusion model; 2) 3D helps 2D in multi-view consistency: the 3D diffusion model enhances the 3D consistency of 2D multi-view sampling process, resulting in more accurate multi-view generation. We validate our idea through extensive experiments in image-based objects and clothed avatar generation tasks. Results show that our method generates realistic 3D objects and avatars with high-fidelity geometry and texture. Extensive ablations also validate our design choices and demonstrate the strong generalization ability to diverse clothing and compositional shapes. Our code and pretrained models will be publicly released on https://yuxuan-xue.com/gen-3diffusion.

翻译：从单张RGB图像创建逼真的3D物体和着装虚拟化身是一个具有吸引力但极具挑战性的问题。由于其不适定性，近期研究利用在大型数据集上预训练的2D扩散模型所蕴含的强大先验知识。尽管2D扩散模型展现出强大的泛化能力，但它们无法保证生成的多视角图像具有3D一致性。本文提出Gen-3Diffusion：通过2D与3D扩散协同实现逼真的图像到3D生成。我们通过精心设计的流程，在训练和采样阶段同步协调预训练的2D扩散模型与3D扩散模型。2D与3D扩散模型的协同作用带来两大优势：1）2D辅助3D泛化：预训练的2D模型对未见图像具有强大的泛化能力，为3D扩散模型提供强形状先验；2）3D辅助2D多视角一致性：3D扩散模型增强2D多视角采样过程的3D一致性，从而生成更精确的多视角图像。我们在基于图像的物体生成和着装虚拟化身生成任务中通过大量实验验证了该方法的有效性。结果表明，我们的方法能够生成具有高保真几何结构与纹理的逼真3D物体与虚拟化身。大量消融实验也验证了我们的设计选择，并证明了对多样化服装与组合形状的强大泛化能力。我们的代码与预训练模型将在 https://yuxuan-xue.com/gen-3diffusion 公开。