Self-supervised Vision Transformers (ViTs) like DINO show an emergent ability to discover objects, typically observed in [CLS] token attention maps of the final layer. However, these maps often contain spurious activations resulting in poor localization of objects. This is because the [CLS] token, trained on an image-level objective, summarizes the entire image instead of focusing on objects. This aggregation dilutes the object-centric information existing in the local, patch-level interactions. We analyze this by computing inter-patch similarity using patch-level attention components (query, key, and value) across all layers. We find that: (1) Object-centric properties are encoded in the similarity maps derived from all three components ($q, k, v$), unlike prior work that uses only key features or the [CLS] token. (2) This object-centric information is distributed across the network, not just confined to the final layer. Based on these insights, we introduce Object-DINO, a training-free method that extracts this distributed object-centric information. Object-DINO clusters attention heads across all layers based on the similarities of their patches and automatically identifies the object-centric cluster corresponding to all objects. We demonstrate Object-DINO's effectiveness on two applications: enhancing unsupervised object discovery (+3.6 to +12.4 CorLoc gains) and mitigating object hallucination in Multimodal Large Language Models by providing visual grounding. Our results demonstrate that using this distributed object-centric information improves downstream tasks without additional training.

翻译：自监督视觉Transformer（如DINO）展现出发现物体的涌现能力，这通常体现在最后一层[CLS]令牌注意力图中。然而，这些图常常包含虚假激活，导致物体定位不佳。这是因为基于图像级目标训练的[CLS]令牌总结了整个图像，而非聚焦于物体。这种聚合稀释了存在于局部补丁级交互中的面向对象信息。我们通过计算跨所有层的补丁级注意力组件（查询、键、值）的补丁间相似性来分析这一点。我们发现：（1）与仅使用键特征或[CLS]令牌的先前工作不同，面向对象属性编码在所有三个组件（$q, k, v$）导出的相似性图中。（2）这种面向对象信息分布在整个网络中，不仅限于最后一层。基于这些洞察，我们引入了Object-DINO，一种无需训练的方法，用于提取这种分布式面向对象信息。Object-DINO根据补丁的相似性对跨所有层的注意力头进行聚类，并自动识别对应所有物体的面向对象聚类。我们在两个应用中展示了Object-DINO的有效性：增强无监督物体发现（+3.6至+12.4 CorLoc增益）以及通过提供视觉定位来缓解多模态大语言模型中的物体幻觉。我们的结果表明，使用这种分布式面向对象信息可在无需额外训练的情况下改进下游任务。