Semantic 2D maps are commonly used by humans and machines for navigation purposes, whether it's walking or driving. However, these maps have limitations: they lack detail, often contain inaccuracies, and are difficult to create and maintain, especially in an automated fashion. Can we use raw imagery to automatically create better maps that can be easily interpreted by both humans and machines? We introduce SNAP, a deep network that learns rich neural 2D maps from ground-level and overhead images. We train our model to align neural maps estimated from different inputs, supervised only with camera poses over tens of millions of StreetView images. SNAP can resolve the location of challenging image queries beyond the reach of traditional methods, outperforming the state of the art in localization by a large margin. Moreover, our neural maps encode not only geometry and appearance but also high-level semantics, discovered without explicit supervision. This enables effective pre-training for data-efficient semantic scene understanding, with the potential to unlock cost-efficient creation of more detailed maps.

翻译：语义二维地图常被人类和机器用于导航任务（无论是步行还是驾驶）。然而，这些地图存在局限性：缺乏细节、常含不精确性，且难以自动创建和维护。能否利用原始图像自动生成能被人类和机器轻松解读的更优地图？我们提出SNAP——一种深度网络，能从地面和俯视图像中学习丰富的神经二维地图。我们训练模型对齐从不同输入估计的神经地图，仅通过数千万张街景图像的相机位姿进行监督。SNAP能解析传统方法难以处理的挑战性图像查询位置，在定位任务上大幅超越现有最优水平。此外，我们的神经地图不仅编码几何与外观信息，还无需显式监督即可发掘高层语义特征。这为数据高效的语义场景理解提供了有效预训练，有望推动更精细地图的低成本创建。