Feedforward models for novel view synthesis (NVS) have recently advanced by transformer-based methods like LVSM, using attention among all input and target views. In this work, we argue that its full self-attention design is suboptimal, suffering from quadratic complexity with respect to the number of input views and rigid parameter sharing among heterogeneous tokens. We propose Efficient-LVSM, a dual-stream architecture that avoids these issues with a decoupled co-refinement mechanism. It applies intra-view self-attention for input views and self-then-cross attention for target views, eliminating unnecessary computation. Efficient-LVSM achieves 29.86 dB PSNR on RealEstate10K with 2 input views, surpassing LVSM by 0.2 dB, with 2x faster training convergence and 4.4x faster inference speed. Efficient-LVSM achieves state-of-the-art performance on multiple benchmarks, exhibits strong zero-shot generalization to unseen view counts, and enables incremental inference with KV-cache, thanks to its decoupled designs.

翻译：用于新视角合成的前馈模型近期通过基于Transformer的方法（如LVSM）取得了进展，这些方法利用了所有输入视图与目标视图之间的注意力机制。本工作中，我们认为其完全自注意力设计并非最优，存在两方面不足：计算复杂度随输入视图数量呈二次方增长，以及异构token间采用僵化的参数共享。我们提出了高效-LVSM，一种双流架构，通过解耦的协同优化机制规避了这些问题。该架构对输入视图采用视图内自注意力，对目标视图采用自注意力后接交叉注意力的方式，从而消除了不必要的计算。高效-LVSM在RealEstate10K数据集上使用2个输入视图时达到29.86 dB的PSNR，超过LVSM 0.2 dB，同时训练收敛速度快2倍，推理速度快4.4倍。高效-LVSM在多个基准测试中取得了最先进的性能，对未见过的视图数量展现出强大的零样本泛化能力，并得益于其解耦设计，能够通过KV缓存实现增量推理。