Novel View Synthesis (NVS), which tries to produce a realistic image at the target view given source view images and their corresponding poses, is a fundamental problem in 3D Vision. As this task is heavily under-constrained, some recent work, like Zero123, tries to solve this problem with generative modeling, specifically using pre-trained diffusion models. Although this strategy generalizes well to new scenes, compared to neural radiance field-based methods, it offers low levels of flexibility. For example, it can only accept a single-view image as input, despite realistic applications often offering multiple input images. This is because the source-view images and corresponding poses are processed separately and injected into the model at different stages. Thus it is not trivial to generalize the model into multi-view source images, once they are available. To solve this issue, we try to process each pose image pair separately and then fuse them as a unified visual representation which will be injected into the model to guide image synthesis at the target-views. However, inconsistency and computation costs increase as the number of input source-view images increases. To solve these issues, the Multi-view Cross Former module is proposed which maps variable-length input data to fix-size output data. A two-stage training strategy is introduced to further improve the efficiency during training time. Qualitative and quantitative evaluation over multiple datasets demonstrates the effectiveness of the proposed method against previous approaches. The code will be released according to the acceptance.

翻译：新视角合成（NVS）旨在根据源视角图像及其对应姿态生成目标视角的真实图像，是三维视觉领域的基础问题。由于该任务严重欠约束，近期研究（如Zero123）尝试通过生成式建模（特别是预训练扩散模型）解决此问题。尽管该策略相比基于神经辐射场的方法具有更好的场景泛化能力，但其灵活性较低：例如，实际应用常提供多幅输入图像，而此类方法仅能接受单视角图像作为输入。这是因为源视角图像与对应姿态被分开处理，并在模型不同阶段注入，导致难以轻易泛化至多源视角场景。为解决此问题，我们尝试分别处理每对姿态-图像，并将其融合为统一视觉表征注入模型以指导目标视角图像合成。然而，随着输入源视角图像数量增加，不一致性与计算开销也随之增长。针对这些挑战，本文提出多视角跨模态融合模块（Multi-view Cross Former），可将变长输入数据映射为定长输出数据，并引入两阶段训练策略进一步提升训练效率。多数据集上的定性与定量评估证明了该方法相较现有技术的有效性。代码将在论文接收后开源。