Many text mining models are constructed by fine-tuning a large deep pre-trained language model (PLM) in downstream tasks. However, a significant challenge nowadays is maintaining performance when we use a lightweight model with limited labelled samples. We present DisCo, a semi-supervised learning (SSL) framework for fine-tuning a cohort of small student models generated from a large PLM using knowledge distillation. Our key insight is to share complementary knowledge among distilled student cohorts to promote their SSL effectiveness. DisCo employs a novel co-training technique to optimize a cohort of multiple small student models by promoting knowledge sharing among students under diversified views: model views produced by different distillation strategies and data views produced by various input augmentations. We evaluate DisCo on both semi-supervised text classification and extractive summarization tasks. Experimental results show that DisCo can produce student models that are 7.6 times smaller and 4.8 times faster in inference than the baseline PLMs while maintaining comparable performance. We also show that DisCo-generated student models outperform the similar-sized models elaborately tuned in distinct tasks.

翻译：许多文本挖掘模型通过在下游任务中对大型深度预训练语言模型（PLM）进行微调而构建。然而，当前一个重大挑战是如何在使用轻量级模型且标注样本有限的情况下保持性能。我们提出DisCo，一个半监督学习（SSL）框架，通过知识蒸馏从大型PLM生成一组小规模学生模型并进行微调。核心见解在于在蒸馏学生群体中共享互补知识以提升其SSL效能。DisCo采用新型协同训练技术，通过促进学生在多样化视角（不同蒸馏策略产生的模型视角和多种输入增强产生的数据视角）下的知识共享来优化多个小规模学生模型组成的群体。我们在半监督文本分类和抽取式摘要任务上评估DisCo。实验结果表明，DisCo生成的学生模型在推理时体积缩小7.6倍、速度提升4.8倍，同时保持与基线PLM相当的性能。我们还显示，DisCo生成的学生模型在不同任务中表现优于精心调优的同等规模模型。