Self-supervised learning (SSL) has made remarkable progress in visual representation learning. Some studies combine SSL with knowledge distillation (SSL-KD) to boost the representation learning performance of small models. In this study, we propose a Multi-mode Online Knowledge Distillation method (MOKD) to boost self-supervised visual representation learning. Different from existing SSL-KD methods that transfer knowledge from a static pre-trained teacher to a student, in MOKD, two different models learn collaboratively in a self-supervised manner. Specifically, MOKD consists of two distillation modes: self-distillation and cross-distillation modes. Among them, self-distillation performs self-supervised learning for each model independently, while cross-distillation realizes knowledge interaction between different models. In cross-distillation, a cross-attention feature search strategy is proposed to enhance the semantic feature alignment between different models. As a result, the two models can absorb knowledge from each other to boost their representation learning performance. Extensive experimental results on different backbones and datasets demonstrate that two heterogeneous models can benefit from MOKD and outperform their independently trained baseline. In addition, MOKD also outperforms existing SSL-KD methods for both the student and teacher models.

翻译：自监督学习（SSL）在视觉表示学习领域取得了显著进展。部分研究将自监督学习与知识蒸馏（SSL-KD）相结合，以提升小模型的表示学习性能。本文提出多模态在线知识蒸馏方法（MOKD），用于增强自监督视觉表示学习。与现有SSL-KD方法从静态预训练教师模型向学生模型传递知识不同，MOKD中两个不同模型以自监督方式协同学习。具体而言，MOKD包含两种蒸馏模式：自蒸馏与交叉蒸馏。其中，自蒸馏使每个模型独立执行自监督学习，交叉蒸馏则实现不同模型间的知识交互。在交叉蒸馏中，我们提出跨注意力特征搜索策略以增强不同模型间的语义特征对齐。由此，两个模型可相互吸收知识以提升表示学习性能。在多种主干网络与数据集上的大量实验结果表明，两个异构模型均能从MOKD中获益，并优于其独立训练的基线模型。此外，MOKD在学生模型与教师模型上的表现均优于现有SSL-KD方法。