How do two sets of images differ? Discerning set-level differences is crucial for understanding model behaviors and analyzing datasets, yet manually sifting through thousands of images is impractical. To aid in this discovery process, we explore the task of automatically describing the differences between two $\textbf{sets}$ of images, which we term Set Difference Captioning. This task takes in image sets $D_A$ and $D_B$, and outputs a description that is more often true on $D_A$ than $D_B$. We outline a two-stage approach that first proposes candidate difference descriptions from image sets and then re-ranks the candidates by checking how well they can differentiate the two sets. We introduce VisDiff, which first captions the images and prompts a language model to propose candidate descriptions, then re-ranks these descriptions using CLIP. To evaluate VisDiff, we collect VisDiffBench, a dataset with 187 paired image sets with ground truth difference descriptions. We apply VisDiff to various domains, such as comparing datasets (e.g., ImageNet vs. ImageNetV2), comparing classification models (e.g., zero-shot CLIP vs. supervised ResNet), summarizing model failure modes (supervised ResNet), characterizing differences between generative models (e.g., StableDiffusionV1 and V2), and discovering what makes images memorable. Using VisDiff, we are able to find interesting and previously unknown differences in datasets and models, demonstrating its utility in revealing nuanced insights.

翻译：两组图像之间有何差异？识别集合层面的差异对于理解模型行为和分析数据集至关重要，但人工筛选数千幅图像并不可行。为辅助这一发现过程，我们探索了自动描述两组$\textbf{图像集}$之间差异的任务，并将其称为集合差异描述（Set Difference Captioning）。该任务输入图像集$D_A$和$D_B$，输出一个在$D_A$上比在$D_B$上更常成立的描述。我们提出了一种两阶段方法：首先从图像集中生成候选差异描述，然后通过检验这些描述对两组图像集的区分能力进行重新排序。我们引入了VisDiff，它首先为图像生成描述并提示语言模型提出候选描述，随后使用CLIP对这些描述进行重新排序。为评估VisDiff，我们构建了VisDiffBench数据集，其中包含187对配有真实差异描述的图像集。我们将VisDiff应用于多个领域，包括对比数据集（如ImageNet与ImageNetV2）、比较分类模型（如零样本CLIP与监督ResNet）、总结模型失效模式（监督ResNet）、刻画生成模型间的差异（如StableDiffusionV1与V2），以及发现图像令人难忘的原因。通过VisDiff，我们能够发现数据集和模型中有趣且先前未知的差异，这证明了其在揭示细微洞察方面的实用价值。