Synthesizing novel view images from a few views is a challenging but practical problem. Existing methods often struggle with producing high-quality results or necessitate per-object optimization in such few-view settings due to the insufficient information provided. In this work, we explore leveraging the strong 2D priors in pre-trained diffusion models for synthesizing novel view images. 2D diffusion models, nevertheless, lack 3D awareness, leading to distorted image synthesis and compromising the identity. To address these problems, we propose DreamSparse, a framework that enables the frozen pre-trained diffusion model to generate geometry and identity-consistent novel view image. Specifically, DreamSparse incorporates a geometry module designed to capture 3D features from sparse views as a 3D prior. Subsequently, a spatial guidance model is introduced to convert these 3D feature maps into spatial information for the generative process. This information is then used to guide the pre-trained diffusion model, enabling it to generate geometrically consistent images without tuning it. Leveraging the strong image priors in the pre-trained diffusion models, DreamSparse is capable of synthesizing high-quality novel views for both object and scene-level images and generalising to open-set images. Experimental results demonstrate that our framework can effectively synthesize novel view images from sparse views and outperforms baselines in both trained and open-set category images. More results can be found on our project page: https://sites.google.com/view/dreamsparse-webpage.

翻译：从少量视图中合成新视角图像是一个具有挑战性且实用的问题。现有方法在稀疏视图设置下，由于信息不足，通常难以生成高质量结果或需要针对每个对象进行优化。在这项工作中，我们探索利用预训练扩散模型中的强二维先验来合成新视角图像。然而，二维扩散模型缺乏三维感知能力，导致图像合成失真并破坏身份一致性。为解决这些问题，我们提出DreamSparse框架，该框架使冻结的预训练扩散模型能够生成几何与身份一致的新视角图像。具体而言，DreamSparse引入一个几何模块，从稀疏视图中捕获三维特征作为三维先验；随后设计空间引导模型，将这些三维特征图转换为生成过程所需的的空间信息，进而引导预训练扩散模型生成几何一致的图像，且无需微调模型。借助预训练扩散模型的强图像先验，DreamSparse能够为物体级和场景级图像合成高质量新视角，并泛化到开放集图像。实验结果表明，该框架可有效从稀疏视图合成新视角图像，且在训练集与开放集类别图像上均优于基准方法。更多结果详见项目页面：https://sites.google.com/view/dreamsparse-webpage。