Self-supervised learning (SSL) pipelines differ in many design choices such as the architecture, augmentations, or pretraining data. Yet SSL is typically evaluated using a single metric: linear probing on ImageNet. This does not provide much insight into why or when a model is better, now how to improve it. To address this, we propose an SSL risk decomposition, which generalizes the classical supervised approximation-estimation decomposition by considering errors arising from the representation learning step. Our decomposition consists of four error components: approximation, representation usability, probe generalization, and encoder generalization. We provide efficient estimators for each component and use them to analyze the effect of 30 design choices on 169 SSL vision models evaluated on ImageNet. Our analysis gives valuable insights for designing and using SSL models. For example, it highlights the main sources of error and shows how to improve SSL in specific settings (full- vs few-shot) by trading off error components. All results and pretrained models are at https://github.com/YannDubs/SSL-Risk-Decomposition.

翻译：自监督学习（SSL）流程在架构、数据增强或预训练数据等设计选择上存在诸多差异。然而，SSL通常仅通过单一指标进行评估：在ImageNet上的线性探针测试。这无法深入揭示模型为何更优、何时更优，也无法指导改进方向。针对这一问题，我们提出了一种SSL风险分解方法，该方法通过考虑表征学习步骤中产生的误差，将经典的监督式逼近-估计分解进行了泛化。我们的分解包含四个误差分量：逼近误差、表征可用性误差、探针泛化误差和编码器泛化误差。我们为每个分量提供了高效估计器，并利用它们分析了30种设计选择对169个在ImageNet上评估的SSL视觉模型的影响。这一分析为设计和使用SSL模型提供了宝贵见解。例如，它揭示了误差的主要来源，并展示了如何在不同设置（全样本vs少样本）下通过权衡误差分量来优化SSL。所有结果及预训练模型均可在https://github.com/YannDubs/SSL-Risk-Decomposition获取。