Text-to-image diffusion models understand spatial relationship between objects, but do they represent the true 3D structure of the world from only 2D supervision? We demonstrate that yes, 3D knowledge is encoded in 2D image diffusion models like Stable Diffusion, and we show that this structure can be exploited for 3D vision tasks. Our method, Viewpoint Neural Textual Inversion (ViewNeTI), controls the 3D viewpoint of objects in generated images from frozen diffusion models. We train a small neural mapper to take camera viewpoint parameters and predict text encoder latents; the latents then condition the diffusion generation process to produce images with the desired camera viewpoint. ViewNeTI naturally addresses Novel View Synthesis (NVS). By leveraging the frozen diffusion model as a prior, we can solve NVS with very few input views; we can even do single-view novel view synthesis. Our single-view NVS predictions have good semantic details and photorealism compared to prior methods. Our approach is well suited for modeling the uncertainty inherent in sparse 3D vision problems because it can efficiently generate diverse samples. Our view-control mechanism is general, and can even change the camera view in images generated by user-defined prompts.

翻译：文本到图像扩散模型能够理解物体间的空间关系，但仅通过二维监督学习，它们能否真正表征世界的三维结构？我们证明，像Stable Diffusion这样的2D图像扩散模型确实编码了三维知识，并且可将其用于三维视觉任务。我们提出的方法——视角神经文本反转（ViewNeTI）——能够控制冻结扩散模型生成图像中物体的三维视角。通过训练一个小型神经映射器，输入相机视角参数并预测文本编码器的隐向量，这些隐向量即可引导扩散生成过程，输出具有指定相机视角的图像。ViewNeTI天然适用于新视角合成（NVS）任务。通过利用冻结扩散模型作为先验，我们仅需极少量输入视角即可实现NVS，甚至能完成单视图新视角合成。与现有方法相比，我们的单视图NVS预测具有更优的语义细节和照片真实感。该方法能高效生成多样化样本，非常适合建模稀疏三维视觉问题中的固有不确定性。此外，我们的视角控制机制具有通用性，甚至可改变用户定义提示词生成图像中的相机视角。