Scene coordinates regression (SCR), i.e., predicting 3D coordinates for every pixel of a given image, has recently shown promising potential. However, existing methods remain mostly scene-specific or limited to small scenes and thus hardly scale to realistic datasets. In this paper, we propose a new paradigm where a single generic SCR model is trained once to be then deployed to new test scenes, regardless of their scale and without further finetuning. For a given query image, it collects inputs from off-the-shelf image retrieval techniques and Structure-from-Motion databases: a list of relevant database images with sparse pointwise 2D-3D annotations. The model is based on the transformer architecture and can take a variable number of images and sparse 2D-3D annotations as input. It is trained on a few diverse datasets and significantly outperforms other scene regression approaches on several benchmarks, including scene-specific models, for visual localization. In particular, we set a new state of the art on the Cambridge localization benchmark, even outperforming feature-matching-based approaches.

翻译：场景坐标回归（SCR）旨在预测给定图像中每个像素的三维坐标，近年来展现出巨大潜力。然而，现有方法大多局限于特定场景或仅适用于小规模场景，难以扩展至实际大型数据集。本文提出一种新范式：通过单一通用SCR模型进行一次性训练，即可部署至任意新测试场景——无论其规模大小，且无需额外微调。对于任意查询图像，该模型从现成的图像检索技术与运动恢复结构数据库中收集输入信息：包含相关数据库图像及其稀疏点状2D-3D标注的列表。模型基于Transformer架构，可接收可变数量的图像与稀疏2D-3D标注作为输入。通过在若干多样化数据集上训练，该模型在多个视觉定位基准测试中显著超越其他场景回归方法（包括场景专用模型）。其中，我们在剑桥定位基准上创下新的最优结果，甚至超越了基于特征匹配的方法。