Text-guided diffusion models have shown superior performance in image/video generation and editing. While few explorations have been performed in 3D scenarios. In this paper, we discuss three fundamental and interesting problems on this topic. First, we equip text-guided diffusion models to achieve 3D-consistent generation. Specifically, we integrate a NeRF-like neural field to generate low-resolution coarse results for a given camera view. Such results can provide 3D priors as condition information for the following diffusion process. During denoising diffusion, we further enhance the 3D consistency by modeling cross-view correspondences with a novel two-stream (corresponding to two different views) asynchronous diffusion process. Second, we study 3D local editing and propose a two-step solution that can generate 360-degree manipulated results by editing an object from a single view. Step 1, we propose to perform 2D local editing by blending the predicted noises. Step 2, we conduct a noise-to-text inversion process that maps 2D blended noises into the view-independent text embedding space. Once the corresponding text embedding is obtained, 360-degree images can be generated. Last but not least, we extend our model to perform one-shot novel view synthesis by fine-tuning on a single image, firstly showing the potential of leveraging text guidance for novel view synthesis. Extensive experiments and various applications show the prowess of our 3DDesigner. The project page is available at https://3ddesigner-diffusion.github.io/.

翻译：文本引导的扩散模型在图像/视频生成和编辑中展现出卓越性能，但在3D场景中的探索尚属鲜见。本文探讨了该领域中三个基础且有趣的问题。首先，我们为文本引导扩散模型赋予实现3D一致性生成的能力。具体而言，我们整合神经辐射场（NeRF）类神经场，为给定相机视角生成低分辨率粗糙结果。此类结果可为后续扩散过程提供3D先验条件信息。在去噪扩散过程中，我们通过新颖的双流（对应两个不同视角）异步扩散过程建模跨视角对应关系，进一步增强3D一致性。其次，我们研究3D局部编辑问题，提出通过单视角目标编辑生成360度操控结果的两步解决方案。第一步，我们通过混合预测噪声实现2D局部编辑。第二步，我们执行噪声到文本嵌入的反演过程，将2D混合噪声映射到与视角无关的文本嵌入空间。获得对应文本嵌入后即可生成360度图像。最后但同样重要的是，我们扩展模型以支持单图像微调后的单次新视角合成，首次展示了利用文本引导进行新视角合成的潜力。大量实验与各类应用验证了3DDesigner的强大能力。项目页面见https://3ddesigner-diffusion.github.io/。