Self-supervised learning (SSL) has achieved notable success in many speech processing tasks, but the large model size and heavy computational cost hinder the deployment. Knowledge distillation trains a small student model to mimic the behavior of a large teacher model. However, the student architecture usually needs to be manually designed and will remain fixed during training, which requires prior knowledge and can lead to suboptimal performance. Inspired by recent success of task-specific structured pruning, we propose DPHuBERT, a novel task-agnostic compression method for speech SSL based on joint distillation and pruning. Experiments on SUPERB show that DPHuBERT outperforms pure distillation methods in almost all tasks. Moreover, DPHuBERT requires little training time and performs well with limited training data, making it suitable for resource-constrained applications. Our method can also be applied to various speech SSL models. Our code and models will be publicly available.

翻译：自监督学习（SSL）已在众多语音处理任务中取得显著成功，但模型参数量庞大且计算成本高昂，阻碍了其实际部署。知识蒸馏通过训练小型学生模型模仿大型教师模型的行为来压缩模型，但学生模型架构通常需要人工设计，并在训练过程中固定不变，这不仅依赖先验知识，还会导致性能次优。受近期任务特定结构化剪枝成功经验的启发，我们提出DPHuBERT——一种基于联合蒸馏与剪枝的语音自监督学习通用压缩方法。在SUPERB基准上的实验表明，DPHuBERT在几乎所有任务上均优于纯蒸馏方法。此外，DPHuBERT仅需极少的训练时间，且能在有限训练数据下保持优异性能，特别适合资源受限的应用场景。该方法还可灵活应用于多种语音自监督学习模型。我们的代码与模型将公开发布。