Vision-Language models (VLMs) that use contrastive language-image pre-training have shown promising zero-shot classification performance. However, their performance on imbalanced dataset is relatively poor, where the distribution of classes in the training dataset is skewed, leading to poor performance in predicting minority classes. For instance, CLIP achieved only 5% accuracy on the iNaturalist18 dataset. We propose to add a lightweight decoder to VLMs to avoid OOM (out of memory) problem caused by large number of classes and capture nuanced features for tail classes. Then, we explore improvements of VLMs using prompt tuning, fine-tuning, and incorporating imbalanced algorithms such as Focal Loss, Balanced SoftMax and Distribution Alignment. Experiments demonstrate that the performance of VLMs can be further boosted when used with decoder and imbalanced methods. Specifically, our improved VLMs significantly outperforms zero-shot classification by an average accuracy of 6.58%, 69.82%, and 6.17%, on ImageNet-LT, iNaturalist18, and Places-LT, respectively. We further analyze the influence of pre-training data size, backbones, and training cost. Our study highlights the significance of imbalanced learning algorithms in face of VLMs pre-trained by huge data. We release our code at https://github.com/Imbalance-VLM/Imbalance-VLM.

翻译：视觉语言模型（VLMs）通过对比语言-图像预训练展现出卓越的零样本分类性能。然而，这些模型在不平衡数据集上的表现相对较差——训练数据中类别分布存在倾斜，导致对少数类别的预测能力不足。例如，CLIP在iNaturalist18数据集上的准确率仅为5%。我们提出在VLM中添加轻量化解码器，以避免因类别数量过多导致的OOM（内存溢出）问题，并捕获尾部类别的细微特征。随后，我们探索通过提示调优、微调，以及引入Focal Loss、Balanced SoftMax和Distribution Alignment等不平衡学习算法来改进VLM。实验表明，当与解码器和不平衡方法结合使用时，VLM的性能可进一步提升。具体而言，改进后的VLM在ImageNet-LT、iNaturalist18和Places-LT数据集上，分别比零样本分类的平均准确率提升了6.58%、69.82%和6.17%。我们进一步分析了预训练数据规模、主干网络和训练成本的影响。本研究强调了在利用海量数据预训练的VLM时，不平衡学习算法的重要性。代码已开源：https://github.com/Imbalance-VLM/Imbalance-VLM。