Retrieval-based language models are increasingly employed in question-answering tasks. These models search in a corpus of documents for relevant information instead of having all factual knowledge stored in its parameters, thereby enhancing efficiency, transparency, and adaptability. We develop the first Norwegian retrieval-based model by adapting the REALM framework and evaluating it on various tasks. After training, we also separate the language model, which we call the reader, from the retriever components, and show that this can be fine-tuned on a range of downstream tasks. Results show that retrieval augmented language modeling improves the reader's performance on extractive question-answering, suggesting that this type of training improves language models' general ability to use context and that this does not happen at the expense of other abilities such as part-of-speech tagging, dependency parsing, named entity recognition, and lemmatization. Code, trained models, and data are made publicly available.

翻译：摘要：基于检索的语言模型越来越多地被应用于问答任务。这类模型通过从文档语料库中搜索相关信息，而非将所有事实知识存储于参数中，从而提升效率、透明度和适应性。我们通过适配REALM框架并评估其在多种任务上的表现，开发了首个挪威语检索增强型模型。训练完成后，我们将语言模型（称为阅读器）与检索器组件分离，并证明该阅读器可在多种下游任务上进行微调。结果表明，检索增强的语言建模能够提升阅读器在抽取式问答任务上的性能，这表明此类训练增强了语言模型利用上下文的通用能力，且不会以牺牲词性标注、依存句法分析、命名实体识别和词形还原等其他能力为代价。相关代码、训练模型及数据均已公开提供。