Vision-language models such as CLIP are capable of mapping the different modality data into a unified feature space, enabling zero/few-shot inference by measuring the similarity of given images and texts. However, most existing methods overlook modality gaps in CLIP's encoded features, which is shown as the text and image features lie far apart from each other, resulting in limited classification performance. To tackle this issue, we introduce a method called Selective Vision-Language Subspace Projection (SSP), which incorporates local image features and utilizes them as a bridge to enhance the alignment between image-text pairs. Specifically, our SSP framework comprises two parallel modules: a vision projector and a language projector. Both projectors utilize local image features to span the respective subspaces for image and texts, thereby projecting the image and text features into their respective subspaces to achieve alignment. Moreover, our approach entails only training-free matrix calculations and can be seamlessly integrated into advanced CLIP-based few-shot learning frameworks. Extensive experiments on 11 datasets have demonstrated SSP's superior text-image alignment capabilities, outperforming the state-of-the-art alignment methods. The code is available at https://github.com/zhuhsingyuu/SSP

翻译：诸如CLIP的视觉-语言模型能够将不同模态数据映射到统一的特征空间，通过度量给定图像与文本的相似性实现零样本/少样本推理。然而，现有方法大多忽视了CLIP编码特征中的模态鸿沟，表现为文本与图像特征彼此相距甚远，导致分类性能受限。为解决此问题，我们提出了一种称为选择性视觉-语言子空间投影（SSP）的方法，该方法引入局部图像特征并将其作为增强图文对对齐的桥梁。具体而言，我们的SSP框架包含两个并行模块：视觉投影器与语言投影器。两个投影器均利用局部图像特征来分别张成图像与文本的子空间，从而将图像和文本特征投影至各自子空间以实现对齐。此外，该方法仅需免训练的矩阵计算，并可无缝集成到先进的基于CLIP的少样本学习框架中。在11个数据集上的大量实验表明，SSP具有卓越的图文对齐能力，其性能优于当前最先进的对齐方法。代码公开于https://github.com/zhuhsingyuu/SSP