We present a new generalizable NeRF method that is able to directly generalize to new unseen scenarios and perform novel view synthesis with as few as two source views. The key to our approach lies in the explicitly modeled correspondence matching information, so as to provide the geometry prior to the prediction of NeRF color and density for volume rendering. The explicit correspondence matching is quantified with the cosine similarity between image features sampled at the 2D projections of a 3D point on different views, which is able to provide reliable cues about the surface geometry. Unlike previous methods where image features are extracted independently for each view, we consider modeling the cross-view interactions via Transformer cross-attention, which greatly improves the feature matching quality. Our method achieves state-of-the-art results on different evaluation settings, with the experiments showing a strong correlation between our learned cosine feature similarity and volume density, demonstrating the effectiveness and superiority of our proposed method. Code is at https://github.com/donydchen/matchnerf

翻译：我们提出了一种新的可泛化NeRF方法，能够直接泛化到未见过的场景，并仅用最少两张源视角图像实现新颖视角合成。该方法的核心在于显式建模的对应匹配信息，从而为神经辐射场的颜色与密度预测提供几何先验，以支持体渲染。显式对应匹配通过在不同视角下三维点二维投影位置采样的图像特征之间的余弦相似度量化，能够提供关于表面几何的可靠线索。与先前方法中每个视角独立提取图像特征的方式不同，我们通过Transformer交叉注意力机制建模跨视角交互，显著提升了特征匹配质量。该方法在不同评估设置下均取得最先进结果，实验表明所学习的余弦特征相似度与体密度之间存在强相关性，验证了所提方法的有效性与优越性。代码地址：https://github.com/donydchen/matchnerf