With the development of deep learning, advanced dialogue generation methods usually require a greater amount of computational resources. One promising approach to obtaining a high-performance and lightweight model is knowledge distillation, which relies heavily on the pre-trained powerful teacher. Collaborative learning, also known as online knowledge distillation, is an effective way to conduct one-stage group distillation in the absence of a well-trained large teacher model. However, previous work has a severe branch homogeneity problem due to the same training objective and the independent identical training sets. To alleviate this problem, we consider the dialogue attributes in the training of network branches. Each branch learns the attribute-related features based on the selected subset. Furthermore, we propose a dual group-based knowledge distillation method, consisting of positive distillation and negative distillation, to further diversify the features of different branches in a steadily and interpretable way. The proposed approach significantly improves branch heterogeneity and outperforms state-of-the-art collaborative learning methods on two widely used open-domain dialogue datasets.

翻译：随着深度学习的发展，先进的对话生成方法通常需要更多的计算资源。获得高性能轻量级模型的一种有前景的方法是知识蒸馏，该方法高度依赖于预训练的强教师模型。协作学习（也称为在线知识蒸馏）是在缺乏训练有素的大型教师模型的情况下进行单阶段组蒸馏的有效方式。然而，先前的工作由于相同的训练目标和独立同分布的训练集而存在严重的分支同质性问题。为缓解该问题，我们考虑了网络分支训练中的对话属性。每个分支基于所选子集学习与属性相关的特征。此外，我们提出了一种基于双组的知识蒸馏方法，包括正蒸馏和负蒸馏，以稳定且可解释的方式进一步多样化不同分支的特征。所提出的方法显著提高了分支异质性，并在两个广泛使用的开放域对话数据集上优于最先进的协作学习方法。