Generalized Category Discovery (GCD) requires a model to both classify known categories and cluster unknown categories in unlabeled data. Prior methods leveraged self-supervised pre-training combined with supervised fine-tuning on the labeled data, followed by simple clustering methods. In this paper, we posit that such methods are still prone to poor performance on out-of-distribution categories, and do not leverage a key ingredient: Semantic relationships between object categories. We therefore propose to leverage multi-modal (vision and language) models, in two complementary ways. First, we establish a strong baseline by replacing uni-modal features with CLIP, inspired by its zero-shot performance. Second, we propose a novel retrieval-based mechanism that leverages CLIP's aligned vision-language representations by mining text descriptions from a text corpus for the labeled and unlabeled set. We specifically use the alignment between CLIP's visual encoding of the image and textual encoding of the corpus to retrieve top-k relevant pieces of text and incorporate their embeddings to perform joint image+text semi-supervised clustering. We perform rigorous experimentation and ablations (including on where to retrieve from, how much to retrieve, and how to combine information), and validate our results on several datasets including out-of-distribution domains, demonstrating state-of-art results.

翻译：广义类别发现（GCD）要求模型同时完成对未标注数据中已知类别的分类和未知类别的聚类。现有方法通常采用自监督预训练结合标注数据上的有监督微调，并辅以简单聚类技术。本文指出，这类方法在处理分布外类别时仍存在性能缺陷，且未能充分利用一个关键要素：目标类别间的语义关联。为此，我们提出以互补方式利用多模态（视觉与语言）模型：首先，受CLIP零样本性能启发，用其替换单模态特征建立强基线；其次，设计一种基于检索的创新机制，通过从文本语料库中挖掘标注集与未标注集的描述文本，充分利用CLIP对齐的视觉-语言表征。具体而言，我们利用CLIP图像视觉编码与语料文本编码的对齐特性，检索前k相关文本片段，融合其嵌入进行图像+文本的半监督联合聚类。通过系统实验与消融分析（包括检索来源、检索数量及信息融合策略），我们在包含分布外域的多数据集上验证了该方法，并取得了最先进结果。