With the development of neural topic models in recent years, topic modelling is playing an increasingly important role in natural language understanding. However, most existing topic models still rely on bag-of-words (BoW) information, either as training input or training target. This limits their ability to capture word order information in documents and causes them to suffer from the out-of-vocabulary (OOV) issue, i.e. they cannot handle unobserved words in new documents. Contextualized word embeddings from pre-trained language models show superiority in the ability of word sense disambiguation and prove to be effective in dealing with OOV words. In this work, we developed a novel neural topic model combining contextualized word embeddings from the pre-trained language model BERT. The model can infer the topic distribution of a document without using any BoW information. In addition, the model can infer the topic distribution of each word in a document directly from the contextualized word embeddings. Experiments on several datasets show that our model outperforms existing topic models in terms of both document classification and topic coherence metrics and can accommodate unseen words from newly arrived documents. Experiments on the NER dataset also show that our model can produce high-quality word topic representations.

翻译：近年来，随着神经主题模型的发展，主题建模在自然语言理解中扮演着日益重要的角色。然而，现有的大多数主题模型仍依赖于词袋（BoW）信息，无论是作为训练输入还是训练目标。这限制了它们捕捉文档中词序信息的能力，并导致它们面临词汇表外（OOV）问题——即无法处理新文档中未观测到的词汇。预训练语言模型生成的上下文词嵌入在词义消歧方面表现出优越性，并已被证明能有效处理OOV词汇。本文开发了一种新颖的神经主题模型，该模型融合了预训练语言模型BERT的上下文词嵌入，可在完全不使用BoW信息的情况下推断文档的主题分布。此外，该模型能直接从上下文词嵌入中推断文档内每个词的主题分布。在多个数据集上的实验表明，我们的模型在文档分类和主题连贯性指标上均优于现有主题模型，并能适应新到文档中的未观测词汇。在NER数据集上的实验还表明，我们的模型能生成高质量的词主题表示。