Knowledge Distillation (KD) compresses computationally expensive pre-trained language models (PLMs) by transferring their knowledge to smaller models, allowing their use in resource-constrained or real-time settings. However, most smaller models fail to surpass the performance of the original larger model, resulting in sacrificing performance to improve inference speed. To address this issue, we propose Co-Training and Co-Distillation (CTCD), a novel framework that improves performance and inference speed together by co-training two models while mutually distilling knowledge. The CTCD framework successfully achieves this based on two significant findings: 1) Distilling knowledge from the smaller model to the larger model during co-training improves the performance of the larger model. 2) The enhanced performance of the larger model further boosts the performance of the smaller model. The CTCD framework shows promise as it can be combined with existing techniques like architecture design or data augmentation, replacing one-way KD methods, to achieve further performance improvement. Extensive ablation studies demonstrate the effectiveness of CTCD, and the small model distilled by CTCD outperforms the original larger model by a significant margin of 1.66 on the GLUE benchmark.

翻译：知识蒸馏（KD）通过将计算昂贵的预训练语言模型（PLM）的知识迁移到更小的模型中进行压缩，使其能在资源受限或实时场景下部署。然而，大多数小型模型难以超越原始大型模型的性能，导致以牺牲性能为代价提升推理速度。为解决此问题，我们提出协同训练与协同蒸馏（CTCD）这一新型框架，通过协同训练两个模型并相互蒸馏知识，同时提升性能与推理速度。CTCD框架基于两项重要发现成功实现此目标：1）在协同训练过程中，将知识从小型模型蒸馏至大型模型可提升大型模型的性能；2）大型模型性能的提升进一步增强了小型模型的性能。该框架可与现有技术（如架构设计或数据增强）结合，替代单向KD方法以取得更优性能提升。大量消融实验证明了CTCD的有效性，经CTCD蒸馏的小型模型在GLUE基准测试中性能显著超越原始大型模型1.66分。