Self-supervised learning has witnessed great progress in vision and NLP; recently, it also attracted much attention to various medical imaging modalities such as X-ray, CT, and MRI. Existing methods mostly focus on building new pretext self-supervision tasks such as reconstruction, orientation, and masking identification according to the properties of medical images. However, the publicly available self-supervision models are not fully exploited. In this paper, we present a powerful yet efficient self-supervision framework for surgical video understanding. Our key insight is to distill knowledge from publicly available models trained on large generic datasets4 to facilitate the self-supervised learning of surgical videos. To this end, we first introduce a semantic-preserving training scheme to obtain our teacher model, which not only contains semantics from the publicly available models, but also can produce accurate knowledge for surgical data. Besides training with only contrastive learning, we also introduce a distillation objective to transfer the rich learned information from the teacher model to self-supervised learning on surgical data. Extensive experiments on two surgical phase recognition benchmarks show that our framework can significantly improve the performance of existing self-supervised learning methods. Notably, our framework demonstrates a compelling advantage under a low-data regime. Our code is available at https://github.com/xmed-lab/DistillingSelf.

翻译：自我监督的学习在视觉和NLP方面取得了巨大进展;最近,它也吸引了对诸如X光、CT和MRI等各种医疗成像模式的极大关注。现有方法主要侧重于建立新的借口自我监督任务,如重建、定向和根据医疗图像的特性进行遮掩身份识别。然而,公开提供的自监督模式没有得到充分开发。在本文中,我们为外科视频理解提出了一个强大而高效的自我监督框架。我们的关键见解是从在大型通用数据集4 上培训的公开模式中提取知识,4 以便利对外科视频进行自我监督学习。为此,我们首先采用语义保存培训计划,以获得我们的教师模式,其中不仅包含从公开的模型中获得的语义学,而且能够产生准确的外科数据知识。除了仅进行对比性学习的培训外,我们还引入了一种蒸馏目标,将从教师模型中学到的丰富信息传输到外科数据自我监督的学习。两个外科阶段识别基准的深入实验表明,我们的框架能够大大改进现有的自我监督/低级学习框架的优势。