Synthesizing novel view images from a few views is a challenging but practical problem. Existing methods often struggle with producing high-quality results or necessitate per-object optimization in such few-view settings due to the insufficient information provided. In this work, we explore leveraging the strong 2D priors in pre-trained diffusion models for synthesizing novel view images. 2D diffusion models, nevertheless, lack 3D awareness, leading to distorted image synthesis and compromising the identity. To address these problems, we propose DreamSparse, a framework that enables the frozen pre-trained diffusion model to generate geometry and identity-consistent novel view image. Specifically, DreamSparse incorporates a geometry module designed to capture 3D features from sparse views as a 3D prior. Subsequently, a spatial guidance model is introduced to convert these 3D feature maps into spatial information for the generative process. This information is then used to guide the pre-trained diffusion model, enabling it to generate geometrically consistent images without tuning it. Leveraging the strong image priors in the pre-trained diffusion models, DreamSparse is capable of synthesizing high-quality novel views for both object and scene-level images and generalising to open-set images. Experimental results demonstrate that our framework can effectively synthesize novel view images from sparse views and outperforms baselines in both trained and open-set category images. More results can be found on our project page: https://sites.google.com/view/dreamsparse-webpage.

翻译：从少量视角合成新视角图像是一个具有挑战性但实用的问题。由于提供的图像信息不足，现有方法在少视角场景下常难以生成高质量结果，或需要对每个对象进行优化。本文探索利用预训练扩散模型中强大的二维先验知识来合成新视角图像。然而，二维扩散模型缺乏三维感知能力，导致合成的图像出现畸变并破坏物体身份一致性。为解决这些问题，我们提出DreamSparse框架，使冻结的预训练扩散模型能够生成几何一致且身份一致的新视角图像。具体而言，DreamSparse引入一个几何模块，用于从稀疏视角中提取三维特征作为三维先验。随后，通过空间引导模型将这些三维特征图转换为生成过程所需的空间信息。该信息进而指导预训练扩散模型，在无需微调的情况下生成几何一致的图像。凭借预训练扩散模型中强大的图像先验，DreamSparse能够为物体级和场景级图像合成高质量新视角，并泛化至开放集图像。实验结果表明，我们的框架可有效从稀疏视角合成新视角图像，在训练集和开放集图像上均优于基线方法。更多结果可访问项目页面：https://sites.google.com/view/dreamsparse-webpage。