Establishing correspondences between image pairs is a long studied problem in computer vision. With recent large-scale foundation models showing strong zero-shot performance on downstream tasks including classification and segmentation, there has been interest in using the internal feature maps of these models for the semantic correspondence task. Recent works observe that features from DINOv2 and Stable Diffusion (SD) are complementary, the former producing accurate but sparse correspondences, while the latter produces spatially consistent correspondences. As a result, current state-of-the-art methods for semantic correspondence involve combining features from both models in an ensemble. While the performance of these methods is impressive, they are computationally expensive, requiring evaluating feature maps from large-scale foundation models. In this work we take a different approach, instead replacing SD features with a superior matching algorithm which is imbued with the desirable spatial consistency property. Specifically, we replace the standard nearest neighbours matching with an optimal transport algorithm that includes a Gromov Wasserstein spatial smoothness prior. We show that we can significantly boost the performance of the DINOv2 baseline, and be competitive and sometimes surpassing state-of-the-art methods using Stable Diffusion features, while being 5--10x more efficient. We make code available at https://github.com/fsnelgar/semantic_matching_gwot .

翻译：在计算机视觉领域，建立图像对之间的对应关系是一个长期研究的问题。随着近期大规模基础模型在分类与分割等下游任务上展现出强大的零样本性能，研究者开始关注利用这些模型的内部特征图来完成语义对应任务。近期工作发现，DINOv2与Stable Diffusion（SD）模型提取的特征具有互补性：前者能产生准确但稀疏的对应关系，而后者则能生成空间一致的对应关系。因此，当前最先进的语义对应方法通常采用集成策略，将两种模型的特征进行融合。尽管这些方法性能优异，但其计算成本高昂，需要从大规模基础模型中提取特征图。本研究采用了一种不同的思路：通过引入具备理想空间一致性特性的更优匹配算法来替代SD特征。具体而言，我们将标准的最近邻匹配替换为一种融合Gromov Wasserstein空间平滑先验的最优传输算法。实验表明，该方法能显著提升DINOv2基线的性能，在保持5-10倍计算效率优势的同时，其性能与当前采用Stable Diffusion特征的最先进方法相当甚至更优。代码已发布于https://github.com/fsnelgar/semantic_matching_gwot。