In this paper, we study a defense against poisoned encoders in SSL called distillation, which is a defense used in supervised learning originally. Distillation aims to distill knowledge from a given model (a.k.a the teacher net) and transfer it to another (a.k.a the student net). Now, we use it to distill benign knowledge from poisoned pre-trained encoders and transfer it to a new encoder, resulting in a clean pre-trained encoder. In particular, we conduct an empirical study on the effectiveness and performance of distillation against poisoned encoders. Using two state-of-the-art backdoor attacks against pre-trained image encoders and four commonly used image classification datasets, our experimental results show that distillation can reduce attack success rate from 80.87% to 27.51% while suffering a 6.35% loss in accuracy. Moreover, we investigate the impact of three core components of distillation on performance: teacher net, student net, and distillation loss. By comparing 4 different teacher nets, 3 student nets, and 6 distillation losses, we find that fine-tuned teacher nets, warm-up-training-based student nets, and attention-based distillation loss perform best, respectively.

翻译：本文研究了一种针对自监督学习中毒编码器的防御方法——蒸馏技术，该技术最初用于监督学习中的防御任务。蒸馏旨在从给定模型（即教师网络）中提取知识并将其迁移至另一模型（即学生网络）。本研究将其应用于从中毒预训练编码器中提取良性知识并迁移至新编码器，从而获得干净的预训练编码器。具体而言，我们通过实验研究了蒸馏技术对抗中毒编码器的有效性与性能表现。采用两种针对预训练图像编码器的最先进后门攻击方法及四个常用图像分类数据集，实验结果表明：蒸馏可将攻击成功率从80.87%降至27.51%，同时仅牺牲6.35%的准确率。此外，我们进一步探究了蒸馏三个核心组件对性能的影响：教师网络、学生网络及蒸馏损失函数。通过对比4种教师网络、3种学生网络及6种蒸馏损失函数，发现微调教师网络、基于预热训练的学生网络以及基于注意力的蒸馏损失函数分别达到最优性能。