Nearest neighbour based methods have proved to be one of the most successful self-supervised learning (SSL) approaches due to their high generalization capabilities. However, their computational efficiency decreases when more than one neighbour is used. In this paper, we propose a novel contrastive SSL approach, which we call All4One, that reduces the distance between neighbour representations using ''centroids'' created through a self-attention mechanism. We use a Centroid Contrasting objective along with single Neighbour Contrasting and Feature Contrasting objectives. Centroids help in learning contextual information from multiple neighbours whereas the neighbour contrast enables learning representations directly from the neighbours and the feature contrast allows learning representations unique to the features. This combination enables All4One to outperform popular instance discrimination approaches by more than 1% on linear classification evaluation for popular benchmark datasets and obtains state-of-the-art (SoTA) results. Finally, we show that All4One is robust towards embedding dimensionalities and augmentations, surpassing NNCLR and Barlow Twins by more than 5% on low dimensionality and weak augmentation settings. The source code would be made available soon.

翻译：基于最近邻的方法因其高泛化能力而被证明是最成功的自监督学习方法之一。然而，当使用多个邻居时，其计算效率会降低。本文提出了一种新颖的对比自监督学习方法——All4One，通过自注意力机制生成的"质心"来缩减邻居表征之间的距离。我们采用质心对比目标、单邻居对比目标与特征对比目标的联合优化：质心有助于从多个邻居中学习上下文信息，邻居对比使模型能直接从邻居学习表征，而特征对比则促进学习特征独有的表征。这一组合使得All4One在主流基准数据集的线性分类评估中比流行的实例判别方法性能提升超过1%，并取得最优结果。最后，我们证明All4One对嵌入维度和数据增强具有鲁棒性：在低维度和弱增强设置下，其性能比NNCLR和Barlow Twins提升超过5%。源代码将很快公开。