Generalized Category Discovery (GCD) aims to identify novel categories in unlabeled data while leveraging a small labeled subset of known classes. Training a parametric classifier solely on image features often leads to overfitting to old classes, and recent multimodal approaches improve performance by incorporating textual information. However, they treat modalities independently and incur high computational cost. We propose SpectralGCD, an efficient and effective multimodal approach to GCD that uses CLIP cross-modal image-concept similarities as a unified cross-modal representation. Each image is expressed as a mixture over semantic concepts from a large task-agnostic dictionary, which anchors learning to explicit semantics and reduces reliance on spurious visual cues. To maintain the semantic quality of representations learned by an efficient student, we introduce Spectral Filtering which exploits a cross-modal covariance matrix over the softmaxed similarities measured by a strong teacher model to automatically retain only relevant concepts from the dictionary. Forward and reverse knowledge distillation from the same teacher ensures that the cross-modal representations of the student remain both semantically sufficient and well-aligned. Across six benchmarks, SpectralGCD delivers accuracy comparable to or significantly superior to state-of-the-art methods at a fraction of the computational cost. The code is publicly available at: https://github.com/miccunifi/SpectralGCD.

翻译：广义类别发现（GCD）旨在利用已知类别的少量标注数据，识别未标注数据中的新类别。仅基于图像特征训练参数化分类器容易导致对旧类别的过拟合，而近期多模态方法通过引入文本信息提升了性能。然而，这些方法独立处理不同模态且计算成本高昂。本文提出SpectralGCD——一种高效且有效的GCD多模态方法，其使用CLIP跨模态图像-概念相似度作为统一的跨模态表示。每幅图像被表示为来自大型任务无关词典的语义概念的混合，从而将学习锚定于显式语义并减少对虚假视觉线索的依赖。为保持高效学生模型所学表示的语义质量，我们引入谱滤波技术：该方法利用强教师模型度量的softmax化相似度构建跨模态协方差矩阵，以自动从词典中仅保留相关概念。通过同一教师模型进行前向与反向知识蒸馏，确保学生的跨模态表示既语义充分又保持良好对齐。在六个基准测试中，SpectralGCD以远低于现有方法的计算成本，取得了与最先进方法相当或显著更优的准确率。代码已公开于：https://github.com/miccunifi/SpectralGCD。