In the field of novel-view synthesis, the necessity of knowing camera poses (e.g., via Structure from Motion) before rendering has been a common practice. However, the consistent acquisition of accurate camera poses remains elusive, and errors in pose extraction can adversely impact the view synthesis process. To address this challenge, we introduce PF-GRT, a new Pose-Free framework for Generalizable Rendering Transformer, eliminating the need for pre-computed camera poses and instead leveraging feature-matching learned directly from data. PF-GRT is parameterized using a local relative coordinate system, where one of the source images is set as the origin. An OmniView Transformer is designed for fusing multi-view cues under the pose-free setting, where unposed-view fusion and origin-centric aggregation are performed. The 3D point feature along target ray is sampled by projecting onto the selected origin plane. The final pixel intensities are modulated and decoded using another Transformer. PF-GRT demonstrates an impressive ability to generalize to new scenes that were not encountered during the training phase, without the need of pre-computing camera poses. Our experiments with zero-shot rendering on the LLFF, RealEstate-10k, Shiny, and Blender datasets reveal that it produces superior quality in generating photo-realistic images. Moreover, it demonstrates robustness against noise in test camera poses. Code is available at https://zhiwenfan.github.io/PF-GRT/.

翻译：在新视角合成领域，渲染前必须获取相机位姿（例如通过运动恢复结构）已成为普遍做法。然而，精确相机位姿的持续获取仍存在困难，且位姿提取中的误差会对视图合成过程产生不利影响。为解决这一挑战，我们提出 PF-GRT——一种用于通用渲染 Transformer 的新型无位姿框架，该框架无需预计算相机位姿，而是利用直接从数据中学习到的特征匹配。PF-GRT 采用局部相对坐标系进行参数化，其中一张源图像被设为原点。我们设计了全向视图 Transformer（OmniView Transformer），用于在无位姿设置下融合多视角线索，执行无位姿视图融合与基于原点的聚合。通过将目标射线上的三维点投影到选定的原点平面上，采样得到对应的点特征。最终像素强度由另一个 Transformer 进行调制与解码。PF-GRT 展现出卓越的泛化能力，能够直接推广到训练阶段未见过的新场景，且无需预计算相机位姿。我们在 LLFF、RealEstate-10k、Shiny 和 Blender 数据集上的零样本渲染实验表明，该方法在生成逼真图像方面具有卓越质量。此外，它还对测试相机位姿中的噪声表现出鲁棒性。代码已开源：https://zhiwenfan.github.io/PF-GRT/。