Self-supervised learning (SSL) has shown impressive results in downstream classification tasks. However, there is limited work in understanding their failure modes and interpreting their learned representations. In this paper, we study the representation space of state-of-the-art self-supervised models including SimCLR, SwaV, MoCo, BYOL, DINO, SimSiam, VICReg and Barlow Twins. Without the use of class label information, we discover discriminative features that correspond to unique physical attributes in images, present mostly in correctly-classified representations. Using these features, we can compress the representation space by up to 40% without significantly affecting linear classification performance. We then propose Self-Supervised Representation Quality Score (or Q-Score), an unsupervised score that can reliably predict if a given sample is likely to be mis-classified during linear evaluation, achieving AUPRC of 91.45 on ImageNet-100 and 78.78 on ImageNet-1K. Q-Score can also be used as a regularization term on pre-trained encoders to remedy low-quality representations. Fine-tuning with Q-Score regularization can boost the linear probing accuracy of SSL models by up to 5.8% on ImageNet-100 and 3.7% on ImageNet-1K compared to their baselines. Finally, using gradient heatmaps and Salient ImageNet masks, we define a metric to quantify the interpretability of each representation. We show that discriminative features are strongly correlated to core attributes and, enhancing these features through Q-score regularization makes SSL representations more interpretable.

翻译：自监督学习（SSL）在下游分类任务中展现出了卓越的性能，然而当前对其失败模式的理解以及所学表征的解读仍十分有限。本文研究了包括SimCLR、SwaV、MoCo、BYOL、DINO、SimSiam、VICReg和Barlow Twins在内的最先进自监督模型的表征空间。在不使用类别标签信息的情况下，我们发现了对应于图像中独特物理属性的判别性特征，这些特征主要存在于正确分类的表征中。利用这些特征，我们可将表征空间压缩至多40%，且对线性分类性能影响甚微。随后，我们提出了自监督表征质量评分（简称Q-Score）——一种无监督评分方法，能够可靠预测线性评估时给定样本是否可能被误分类，在ImageNet-100和ImageNet-1K上的AUPRC分别达到91.45和78.78。Q-Score还可作为预训练编码器的正则化项，用于修复低质量表征。与基线模型相比，采用Q-Score正则化进行微调可将SSL模型的线性探测准确率在ImageNet-100上提升至多5.8%，在ImageNet-1K上提升3.7%。最后，利用梯度热力图和Salient ImageNet掩码，我们定义了一个指标来量化每个表征的可解释性。研究表明，判别性特征与核心属性高度相关，而通过Q-Score正则化增强这些特征可使SSL表征更具可解释性。