Knowledge Distillation (KD) compresses computationally expensive pre-trained language models (PLMs) by transferring their knowledge to smaller models, allowing their use in resource-constrained or real-time settings. However, most smaller models fail to surpass the performance of the original larger model, resulting in sacrificing performance to improve inference speed. To address this issue, we propose Co-Training and Co-Distillation (CTCD), a novel framework that improves performance and inference speed together by co-training two models while mutually distilling knowledge. The CTCD framework successfully achieves this based on two significant findings: 1) Distilling knowledge from the smaller model to the larger model during co-training improves the performance of the larger model. 2) The enhanced performance of the larger model further boosts the performance of the smaller model. The CTCD framework shows promise as it can be combined with existing techniques like architecture design or data augmentation, replacing one-way KD methods, to achieve further performance improvement. Extensive ablation studies demonstrate the effectiveness of CTCD, and the small model distilled by CTCD outperforms the original larger model by a significant margin of 1.66 on the GLUE benchmark.

翻译：知识蒸馏（KD）通过将计算昂贵的预训练语言模型（PLM）的知识迁移至更小的模型，使其能够在资源受限或实时场景中应用，从而实现对模型的压缩。然而，多数小型模型难以超越原始大型模型的性能表现，导致以牺牲性能为代价换取推理速度提升。为解决这一问题，我们提出协同训练与协同蒸馏（CTCD）框架——一种通过同时训练两个模型并在二者间相互蒸馏知识来同步提升性能与推理速度的新型框架。CTCD框架的可行性基于两项重要发现：1）在协同训练过程中，将知识从较小模型蒸馏至较大模型可提升后者的性能；2）较大模型性能的提升会进一步促进较小模型性能的增长。该框架具有广阔前景，可与现有技术（如架构设计或数据增强）结合，替代单向KD方法以实现更显著的性能提升。大量消融实验证明了CTCD的有效性，经CTCD蒸馏的小型模型在GLUE基准测试中较原始大型模型性能提升高达1.66个点。