Cross-modal distillation has been widely used to transfer knowledge across different modalities, enriching the representation of the target unimodal one. Recent studies highly relate the temporal synchronization between vision and sound to the semantic consistency for cross-modal distillation. However, such semantic consistency from the synchronization is hard to guarantee in unconstrained videos, due to the irrelevant modality noise and differentiated semantic correlation. To this end, we first propose a \textit{Modality Noise Filter} (MNF) module to erase the irrelevant noise in teacher modality with cross-modal context. After this purification, we then design a \textit{Contrastive Semantic Calibration} (CSC) module to adaptively distill useful knowledge for target modality, by referring to the differentiated sample-wise semantic correlation in a contrastive fashion. Extensive experiments show that our method could bring a performance boost compared with other distillation methods in both visual action recognition and video retrieval task. We also extend to the audio tagging task to prove the generalization of our method. The source code is available at \href{https://github.com/GeWu-Lab/cross-modal-distillation}{https://github.com/GeWu-Lab/cross-modal-distillation}.

翻译：跨模态蒸馏已被广泛用于跨不同模态的知识迁移，以丰富目标单模态的表征。近期研究将视觉与声音的时间同步性高度关联于跨模态蒸馏中的语义一致性。然而，在非约束视频中，由于无关模态噪声和差异化语义关联的存在，这种源于同步性的语义一致性难以保证。为此，我们首先提出一种\textit{模态噪声过滤器}（MNF）模块，利用跨模态上下文消除教师模态中的无关噪声。完成净化后，我们进一步设计\textit{对比语义校准}（CSC）模块，通过对比方式参照样本层面的差异化语义关联，自适应蒸馏出目标模态的有用知识。大量实验表明，在视觉动作识别与视频检索任务中，我们的方法相比其他蒸馏方法能带来性能提升。我们还将方法扩展到音频标注任务，以证明其泛化能力。源代码已开源至\href{https://github.com/GeWu-Lab/cross-modal-distillation}{https://github.com/GeWu-Lab/cross-modal-distillation}。