Based on the success of large-scale visual foundation models like CLIP in various downstream tasks, this paper initially attempts to explore their impact on Long-Tailed Semi-Supervised Learning (LTSSL) by employing the foundation model with three strategies: Linear Probing (LP), Lightweight Fine-Tuning (LFT), and Full Fine-Tuning (FFT). Our analysis presents the following insights: i) Compared to LTSSL algorithms trained from scratch, FFT results in a decline in model performance, whereas LP and LFT, although boosting overall model performance, exhibit negligible benefits to tail classes. ii) LP produces numerous false pseudo-labels due to \textit{underlearned} training data, while LFT can reduce the number of these false labels but becomes overconfident about them owing to \textit{biased fitting} training data. This exacerbates the pseudo-labeled and classifier biases inherent in LTSSL, limiting performance improvement in the tail classes. With these insights, we propose a Unbiased Lightweight Fine-tuning strategy, \textbf{ULFine}, which mitigates the overconfidence via confidence-aware adaptive fitting of textual prototypes and counteracts the pseudo-labeled and classifier biases via complementary fusion of dual logits. Extensive experiments demonstrate that ULFine markedly decreases training costs by over ten times and substantially increases prediction accuracies compared to state-of-the-art methods.

翻译：基于CLIP等大规模视觉基础模型在各类下游任务中的成功应用，本文首次尝试探究其对长尾半监督学习的影响，通过采用线性探测、轻量微调与全参数微调三种策略部署基础模型。我们的分析得出以下见解：i) 与从头训练的长尾半监督学习算法相比，全参数微调会导致模型性能下降，而线性探测与轻量微调虽能提升模型整体性能，但对尾部类别的增益微乎其微。ii) 线性探测因训练数据\textit{欠学习}而产生大量错误伪标签，而轻量微调虽能减少此类错误标签数量，却因训练数据的\textit{偏置拟合}而对伪标签过度自信。这加剧了长尾半监督学习中固有的伪标签偏置与分类器偏置，限制了尾部类别的性能提升。基于这些发现，我们提出一种无偏轻量微调策略\textbf{ULFine}：该方法通过文本原型的置信度自适应拟合缓解过度自信问题，并借助双对数输出的互补融合抵消伪标签与分类器偏置。大量实验表明，相较于现有最优方法，ULFine能显著降低十倍以上的训练成本，并大幅提升预测准确率。