Given unlabelled datasets containing both old and new categories, generalized category discovery (GCD) aims to accurately discover new classes while correctly classifying old classes, leveraging the class concepts learned from labeled samples. Current GCD methods only use a single visual modality of information, resulting in poor classification of visually similar classes. As a different modality, text information can provide complementary discriminative information, which motivates us to introduce it into the GCD task. However, the lack of class names for unlabelled data makes it impractical to utilize text information. To tackle this challenging problem, in this paper, we propose a Text Embedding Synthesizer (TES) to generate pseudo text embeddings for unlabelled samples. Specifically, our TES leverages the property that CLIP can generate aligned vision-language features, converting visual embeddings into tokens of the CLIP's text encoder to generate pseudo text embeddings. Besides, we employ a dual-branch framework, through the joint learning and instance consistency of different modality branches, visual and semantic information mutually enhance each other, promoting the interaction and fusion of visual and text knowledge. Our method unlocks the multi-modal potentials of CLIP and outperforms the baseline methods by a large margin on all GCD benchmarks, achieving new state-of-the-art. The code will be released at https://github.com/enguangW/GET .

翻译：给定包含新旧类别的未标记数据集，广义类别发现（GCD）旨在利用从标记样本中习得的类别概念，准确发现新类别并正确分类旧类别。现有GCD方法仅利用单一视觉模态信息，导致视觉相似类别的分类效果不佳。作为一种不同模态，文本信息能够提供互补的判别性信息，这促使我们将其引入GCD任务。然而，未标记数据缺乏类别名称使得直接利用文本信息不可行。针对这一挑战性问题，本文提出文本嵌入合成器（TES）为未标记样本生成伪文本嵌入。具体而言，我们的TES利用CLIP能够生成对齐的视觉-语言特征这一特性，将视觉嵌入转换为CLIP文本编码器的标记以生成伪文本嵌入。此外，我们采用双分支框架，通过不同模态分支的联合学习与实例一致性，视觉信息与语义信息相互增强，促进视觉知识与文本知识的交互融合。我们的方法释放了CLIP的多模态潜力，在所有GCD基准测试中大幅超越基线方法，实现了新的最优性能。代码将在https://github.com/enguangW/GET发布。