Sequence-level knowledge distillation reduces the size of Seq2Seq models for more efficient abstractive summarization. However, it often leads to a loss of abstractiveness in summarization. In this paper, we propose a novel approach named DisCal to enhance the level of abstractiveness (measured by n-gram overlap) without sacrificing the informativeness (measured by ROUGE) of generated summaries. DisCal exposes diverse pseudo summaries with two supervision to the student model. Firstly, the best pseudo summary is identified in terms of abstractiveness and informativeness and used for sequence-level distillation. Secondly, their ranks are used to ensure the student model to assign higher prediction scores to summaries with higher ranks. Our experiments show that DisCal outperforms prior methods in abstractive summarization distillation, producing highly abstractive and informative summaries.

翻译：序列级知识蒸馏能够减小序列到序列（Seq2Seq）模型的规模，从而提高抽象式摘要的生成效率。然而，该方法常导致摘要的抽象性下降。本文提出了一种名为DisCal的新方法，旨在不牺牲生成摘要的信息量（以ROUGE评估）的前提下，提升摘要的抽象性水平（以n元组重叠度衡量）。DisCal通过两种监督方式向学生模型呈现多样的伪摘要。首先，根据抽象性和信息量筛选出最优伪摘要，用于序列级蒸馏。其次，利用伪摘要的排序，确保学生模型为排序更高的摘要分配更高的预测分数。实验结果表明，DisCal在抽象式摘要蒸馏任务中优于现有方法，能够生成兼具高抽象性与高信息量的摘要。