The goal of this paper is to detect what has changed, if anything, between two "in the wild" images of the same 3D scene acquired from different camera positions and at different temporal instances. The open-set nature of this problem, occlusions/dis-occlusions due to the shift in viewpoint, and the lack of suitable training datasets, presents substantial challenges in devising a solution. To address this problem, we contribute a change detection model that is trained entirely on synthetic data and is class-agnostic, yet it is performant out-of-the-box on real world images without requiring fine-tuning. Our solution entails a "register and difference" approach that leverages self-supervised frozen embeddings and feature differences, which allows the model to generalise to a wide variety of scenes and domains. The model is able to operate directly on two RGB images, without requiring access to ground truth camera intrinsics, extrinsics, depth maps, point clouds, or additional before-after images. Finally, we collect and release a new evaluation dataset consisting of real-world image pairs with human-annotated differences and demonstrate the efficacy of our method. The code, datasets and pre-trained model can be found at: https://github.com/ragavsachdeva/CYWS-3D
翻译:本文旨在检测同一三维场景的两张“野外”图像之间是否存在任何变化,这两张图像分别从不同相机位置和不同时间点获取。该问题的开放集特性、视角变化导致的遮挡/遮挡解除,以及缺乏合适的训练数据集,给解决方案的设计带来了重大挑战。为解决这一问题,我们提出了一种基于完全合成数据训练的变化检测模型,该模型与类别无关,且无需微调即可直接应用于真实世界图像并表现出色。我们的方案采用“配准与差分”方法,利用自监督冻结嵌入与特征差异,使模型能够泛化至各种场景与领域。该模型可直接处理两张RGB图像,无需真实相机内参、外参、深度图、点云或额外的前后图像。最后,我们收集并发布了一个新的评估数据集,其中包含人工标注差异的真实世界图像对,并展示了我们方法的有效性。代码、数据集及预训练模型可在 https://github.com/ragavsachdeva/CYWS-3D 获取。