Causal language modeling (LM) uses word history to predict the next word. BERT, on the other hand, makes use of bi-directional word information in a sentence to predict words at masked positions. While BERT is effective in sequence encoding, it is non-causal by nature and is not designed for sequence generation. In this paper, we propose a novel language model, SUffix REtrieval-Augmented LM (SUREALM), that simulates a bi-directional contextual effect in an autoregressive manner. SUREALM employs an embedding retriever to search for training sentences in a data store that share similar word history during sequence generation. In particular, the suffix portions of the retrieved sentences mimick the "future" context. We evaluated our proposed model on the DSTC9 spoken dialogue corpus and showed promising word perplexity reduction on the validation and test set compared to competitive baselines.

翻译：因果语言建模（LM）利用单词历史来预测下一个单词。而BERT则利用句子中的双向单词信息来预测掩蔽位置的单词。尽管BERT在序列编码方面效果显著，但其本质上非因果，且并非为序列生成而设计。本文提出了一种新型语言模型——SUffix REtrieval-Augmented LM（SUREALM），该模型以自回归方式模拟双向语境效应。SUREALM采用嵌入检索器，在数据存储中搜索序列生成过程中共享相似单词历史的训练句子。特别地，所检索句子的后缀部分模拟了"未来"语境。我们在DSTC9口语对话语料库上评估了所提出的模型，与具有竞争力的基线模型相比，在验证集和测试集上均显示出有前景的词困惑度降低效果。