Scene coordinates regression (SCR), i.e., predicting 3D coordinates for every pixel of a given image, has recently shown promising potential. However, existing methods remain mostly scene-specific or limited to small scenes and thus hardly scale to realistic datasets. In this paper, we propose a new paradigm where a single generic SCR model is trained once to be then deployed to new test scenes, regardless of their scale and without further finetuning. For a given query image, it collects inputs from off-the-shelf image retrieval techniques and Structure-from-Motion databases: a list of relevant database images with sparse pointwise 2D-3D annotations. The model is based on the transformer architecture and can take a variable number of images and sparse 2D-3D annotations as input. It is trained on a few diverse datasets and significantly outperforms other scene regression approaches on several benchmarks, including scene-specific models, for visual localization. In particular, we set a new state of the art on the Cambridge localization benchmark, even outperforming feature-matching-based approaches.

翻译：场景坐标回归（SCR）——即为给定图像的每个像素预测三维坐标——近期展现出显著潜力。然而，现有方法大多局限于特定场景或仅适用于小尺度场景，难以扩展至真实数据集。本文提出一种新范式：训练单个通用SCR模型后，无需微调即可直接应用于任意规模的新测试场景。对于查询图像，模型可从现成的图像检索技术与运动恢复结构数据库中获取输入：包含稀疏点状2D-3D标注的相关数据库图像列表。该模型基于Transformer架构，可接受可变数量的图像与稀疏2D-3D标注作为输入。在少量多样化数据集上训练后，该模型在多个基准测试中显著超越其他场景回归方法（包括场景特定模型），尤其在视觉定位任务上。我们在剑桥定位基准上刷新了最先进水平，甚至优于基于特征匹配的方法。