We present a simple but effective pixel-level self-supervised distillation framework friendly to dense prediction tasks. Our method, called Pixel-Wise Contrastive Distillation (PCD), distills knowledge by attracting the corresponding pixels from student's and teacher's output feature maps. PCD includes a novel design called SpatialAdaptor which ``reshapes'' a part of the teacher network while preserving the distribution of its output features. Our ablation experiments suggest that this reshaping behavior enables more informative pixel-to-pixel distillation. Moreover, we utilize a plug-in multi-head self-attention module that explicitly relates the pixels of student's feature maps to enhance the effective receptive field, leading to a more competitive student. PCD \textbf{outperforms} previous self-supervised distillation methods on various dense prediction tasks. A backbone of \mbox{ResNet-18-FPN} distilled by PCD achieves $37.4$ AP$^\text{bbox}$ and $34.0$ AP$^\text{mask}$ on COCO dataset using the detector of \mbox{Mask R-CNN}. We hope our study will inspire future research on how to pre-train a small model friendly to dense prediction tasks in a self-supervised fashion.

翻译：我们提出了一种简单但有效的像素级自监督蒸馏框架，适用于密集预测任务。该方法名为像素级对比蒸馏（PCD），通过吸引学生模型与教师模型输出特征图中对应位置的像素，实现知识蒸馏。PCD包含一个名为SpatialAdaptor的新型设计，它在保持教师网络输出特征分布的同时，对其部分结构进行“重塑”。消融实验表明，这种重塑行为能够实现信息更丰富的逐像素蒸馏。此外，我们利用一个可插拔的多头自注意力模块，显式关联学生特征图中的像素，从而增强有效感受野，使学生模型更具竞争力。PCD在多种密集预测任务上均优于以往的自监督蒸馏方法。使用Mask R-CNN检测器时，经PCD蒸馏的ResNet-18-FPN骨干网络在COCO数据集上达到了37.4 AP$^\text{bbox}$和34.0 AP$^\text{mask}$。我们期望本研究能启发未来关于如何以自监督方式预训练适用于密集预测任务的小模型的研究。