We introduce Zero-1-to-3, a framework for changing the camera viewpoint of an object given just a single RGB image. To perform novel view synthesis in this under-constrained setting, we capitalize on the geometric priors that large-scale diffusion models learn about natural images. Our conditional diffusion model uses a synthetic dataset to learn controls of the relative camera viewpoint, which allow new images to be generated of the same object under a specified camera transformation. Even though it is trained on a synthetic dataset, our model retains a strong zero-shot generalization ability to out-of-distribution datasets as well as in-the-wild images, including impressionist paintings. Our viewpoint-conditioned diffusion approach can further be used for the task of 3D reconstruction from a single image. Qualitative and quantitative experiments show that our method significantly outperforms state-of-the-art single-view 3D reconstruction and novel view synthesis models by leveraging Internet-scale pre-training.

翻译：我们提出Zero-1-to-3框架，该框架仅需单张RGB图像即可改变物体的相机视角。为在这种欠约束设定下实现新视角合成，我们利用大规模扩散模型从自然图像中习得的几何先验知识。我们的条件扩散模型通过合成数据集学习相对相机视角的控制，从而能够生成同一物体在指定相机变换下的新图像。尽管仅在合成数据集上训练，该模型仍保留了对分布外数据集以及包括印象派画作在内的野外图像的强零样本泛化能力。基于视角条件的扩散方法还可进一步用于单图像三维重建任务。定性与定量实验表明，通过利用互联网规模预训练，我们的方法显著优于现有最先进的单视图三维重建与新视角合成模型。