Topic modeling is a dominant method for exploring document collections on the web and in digital libraries. Recent approaches to topic modeling use pretrained contextualized language models and variational autoencoders. However, large neural topic models have a considerable memory footprint. In this paper, we propose a knowledge distillation framework to compress a contextualized topic model without loss in topic quality. In particular, the proposed distillation objective is to minimize the cross-entropy of the soft labels produced by the teacher and the student models, as well as to minimize the squared 2-Wasserstein distance between the latent distributions learned by the two models. Experiments on two publicly available datasets show that the student trained with knowledge distillation achieves topic coherence much higher than that of the original student model, and even surpasses the teacher while containing far fewer parameters than the teacher's. The distilled model also outperforms several other competitive topic models on topic coherence.

翻译：主题建模是探索网络与数字图书馆中文档集合的主流方法。近年来的主题建模方法采用预训练上下文语言模型和变分自编码器。然而，大型神经主题模型存在显著的内存开销。本文提出一种知识蒸馏框架，在保证主题质量不损失的前提下压缩上下文主题模型。具体而言，所提出的蒸馏目标是最小化教师模型与学生模型生成的软标签之间的交叉熵，同时最小化两个模型所学习隐分布之间的平方2-Wasserstein距离。在两个公开数据集上的实验表明，通过知识蒸馏训练的学生模型在主题连贯性上远优于原始学生模型，甚至在参数量远少于教师模型的情况下超越了教师模型。此外，蒸馏后的模型在主题连贯性上优于其他多个具有竞争力的主题模型。