Encouraged by the growing availability of pre-trained 2D diffusion models, image-to-3D generation by leveraging Score Distillation Sampling (SDS) is making remarkable progress. Most existing methods combine novel-view lifting from 2D diffusion models which usually take the reference image as a condition while applying hard L2 image supervision at the reference view. Yet heavily adhering to the image is prone to corrupting the inductive knowledge of the 2D diffusion model leading to flat or distorted 3D generation frequently. In this work, we reexamine image-to-3D in a novel perspective and present Isotropic3D, an image-to-3D generation pipeline that takes only an image CLIP embedding as input. Isotropic3D allows the optimization to be isotropic w.r.t. the azimuth angle by solely resting on the SDS loss. The core of our framework lies in a two-stage diffusion model fine-tuning. Firstly, we fine-tune a text-to-3D diffusion model by substituting its text encoder with an image encoder, by which the model preliminarily acquires image-to-image capabilities. Secondly, we perform fine-tuning using our Explicit Multi-view Attention (EMA) which combines noisy multi-view images with the noise-free reference image as an explicit condition. CLIP embedding is sent to the diffusion model throughout the whole process while reference images are discarded once after fine-tuning. As a result, with a single image CLIP embedding, Isotropic3D is capable of generating multi-view mutually consistent images and also a 3D model with more symmetrical and neat content, well-proportioned geometry, rich colored texture, and less distortion compared with existing image-to-3D methods while still preserving the similarity to the reference image to a large extent. The project page is available at https://isotropic3d.github.io/. The code and models are available at https://github.com/pkunliu/Isotropic3D.

翻译：受预训练二维扩散模型日益普及的鼓舞，利用分数蒸馏采样（SDS）从图像到三维的生成取得了显著进展。现有方法大多结合来自二维扩散模型的新视角提升，通常以参考图像为条件，并在参考视角施加严格的L2图像监督。然而，过度依赖图像易破坏二维扩散模型的归纳知识，导致频繁生成扁平或扭曲的三维模型。在本工作中，我们从全新视角重新审视图像到三维生成，提出Isotropic3D——一种仅以图像CLIP嵌入作为输入的图像到三维生成流水线。Isotropic3D通过仅依赖SDS损失，使得优化过程相对于方位角具有各向同性。我们的框架核心在于两阶段扩散模型微调：首先，通过将文本编码器替换为图像编码器微调文本到三维扩散模型，使模型初步获得图像到图像能力；其次，使用显式多视角注意力（EMA）进行微调，该方法将含噪多视角图像与无噪参考图像结合作为显式条件。整个过程中CLIP嵌入被送入扩散模型，而参考图像在微调完成后被丢弃。因此，与现有图像到三维方法相比，Isotropic3D凭借单个图像CLIP嵌入，不仅能生成多视角一致的图像，还能生成内容更对称整洁、几何比例协调、彩色纹理丰富且畸变更少的三维模型，同时仍能较大程度保持与参考图像的相似性。项目页面：https://isotropic3d.github.io/。代码与模型：https://github.com/pkunliu/Isotropic3D。