Current perceptual similarity metrics operate at the level of pixels and patches. These metrics compare images in terms of their low-level colors and textures, but fail to capture mid-level similarities and differences in image layout, object pose, and semantic content. In this paper, we develop a perceptual metric that assesses images holistically. Our first step is to collect a new dataset of human similarity judgments over image pairs that are alike in diverse ways. Critical to this dataset is that judgments are nearly automatic and shared by all observers. To achieve this we use recent text-to-image models to create synthetic pairs that are perturbed along various dimensions. We observe that popular perceptual metrics fall short of explaining our new data, and we introduce a new metric, DreamSim, tuned to better align with human perception. We analyze how our metric is affected by different visual attributes, and find that it focuses heavily on foreground objects and semantic content while also being sensitive to color and layout. Notably, despite being trained on synthetic data, our metric generalizes to real images, giving strong results on retrieval and reconstruction tasks. Furthermore, our metric outperforms both prior learned metrics and recent large vision models on these tasks.

翻译：当前的感知相似性度量方法在像素和图像块层面运作。这些度量通过比较图像的低级颜色和纹理来评估相似性，但未能捕捉到图像布局、物体姿态和语义内容等中级相似性与差异。本文旨在开发一种从整体角度评估图像的感知度量指标。第一步，我们收集了一个新的数据集，其中包含人类对多样化相似图像对的判断。该数据集的构建关键，在于确保这些判断近乎自动且为所有观察者共享。为此，我们利用最新的文本到图像模型生成沿不同维度扰动的合成图像对。我们观察到，现有流行的感知度量指标无法充分解释新数据，因此提出了一种名为DreamSim的新指标，其调优方向是更好地与人类感知对齐。我们分析了不同视觉属性对DreamSim的影响，发现它高度关注前景物体和语义内容，同时对颜色和布局也保持敏感。值得注意的是，尽管DreamSim基于合成数据训练，但它能泛化至真实图像，在检索和重建任务中展现出优异性能。此外，该指标在此类任务上的表现超越了先前学习的度量指标及近期的大规模视觉模型。