Self-supervised learning is one of the most promising approaches to acquiring knowledge from limited labeled data. Despite the substantial advancements made in recent years, self-supervised models have posed a challenge to practitioners, as they do not readily provide insight into the model's confidence and uncertainty. Tackling this issue is no simple feat, primarily due to the complexity involved in implementing techniques that can make use of the latent representations learned during pre-training without relying on explicit labels. Motivated by this, we introduce a new stochastic vision transformer that integrates uncertainty and distance awareness into self-supervised learning (SSL) pipelines. Instead of the conventional deterministic vector embedding, our novel stochastic vision transformer encodes image patches into elliptical Gaussian distributional embeddings. Notably, the attention matrices of these stochastic representational embeddings are computed using Wasserstein distance-based attention, effectively capitalizing on the distributional nature of these embeddings. Additionally, we propose a regularization term based on Wasserstein distance for both pre-training and fine-tuning processes, thereby incorporating distance awareness into latent representations. We perform extensive experiments across different tasks such as in-distribution generalization, out-of-distribution detection, dataset corruption, semi-supervised settings, and transfer learning to other datasets and tasks. Our proposed method achieves superior accuracy and calibration, surpassing the self-supervised baseline in a wide range of experiments on a variety of datasets.

翻译：自监督学习是从有限标注数据中获取知识的最有前景的方法之一。尽管近年来取得了显著进展，但自监督模型给实践者带来了挑战，因为它们无法直接提供模型置信度和不确定性的见解。解决这一问题并非易事，主要原因是实现能够利用预训练期间学到的潜在表示而不依赖显式标签的技术具有复杂性。受此启发，我们提出了一种新型随机视觉Transformer，该模型将不确定性与距离感知能力融入自监督学习（SSL）流程中。不同于传统的确定性向量嵌入，我们的新型随机视觉Transformer将图像块编码为椭圆高斯分布嵌入。值得注意的是，这些随机表征嵌入的注意力矩阵采用基于Wasserstein距离的注意力机制计算，有效利用了嵌入的分布特性。此外，我们针对预训练和微调过程提出了基于Wasserstein距离的正则化项，从而将距离感知能力引入潜在表示中。我们在分布内泛化、分布外检测、数据集破坏、半监督设置以及向其他数据集和任务的迁移学习等不同任务上进行了广泛实验。所提出的方法在多种数据集的大量实验中实现了优于自监督基线的准确率和校准性能。