In recent years, self-supervised learning (SSL) has emerged as a promising approach for extracting valuable representations from unlabeled data. One successful SSL method is contrastive learning, which aims to bring positive examples closer while pushing negative examples apart. Many current contrastive learning approaches utilize a parameterized projection head. Through a combination of empirical analysis and theoretical investigation, we provide insights into the internal mechanisms of the projection head and its relationship with the phenomenon of dimensional collapse. Our findings demonstrate that the projection head enhances the quality of representations by performing contrastive loss in a projected subspace. Therefore, we propose an assumption that only a subset of features is necessary when minimizing the contrastive loss of a mini-batch of data. Theoretical analysis further suggests that a sparse projection head can enhance generalization, leading us to introduce SparseHead - a regularization term that effectively constrains the sparsity of the projection head, and can be seamlessly integrated with any self-supervised learning (SSL) approaches. Our experimental results validate the effectiveness of SparseHead, demonstrating its ability to improve the performance of existing contrastive methods.

翻译：近年来，自监督学习（SSL）已成为从无标签数据中提取有效表征的一种有前景的方法。其中一种成功的SSL方法是对比学习，其目标是在拉近正样本的同时推开负样本。当前许多对比学习方法都采用参数化的投影头。通过结合实证分析与理论探究，我们深入揭示了投影头的内部机制及其与维度塌缩现象的关系。研究结果表明，投影头通过在投影子空间中执行对比损失来提升表征质量。基于此，我们提出一个假设：在最小化小批量数据的对比损失时，仅需利用特征子集即可。进一步的理论分析表明，稀疏投影头能够增强泛化能力，由此我们引入SparseHead——一种有效约束投影头稀疏性的正则化项，可无缝集成于任何自监督学习方法中。实验结果验证了SparseHead的有效性，证明其能够提升现有对比方法的性能。