Machine Reading Comprehension (MRC) has been a long-standing problem in NLP and, with the recent introduction of the BERT family of transformer based language models, it has come a long way to getting solved. Unfortunately, however, when BERT variants trained on general text corpora are applied to domain-specific text, their performance inevitably degrades on account of the domain shift i.e. genre/subject matter discrepancy between the training and downstream application data. Knowledge graphs act as reservoirs for either open or closed domain information and prior studies have shown that they can be used to improve the performance of general-purpose transformers in domain-specific applications. Building on existing work, we introduce a method using Multi-Layer Perceptrons (MLPs) for aligning and integrating embeddings extracted from knowledge graphs with the embeddings spaces of pre-trained language models (LMs). We fuse the aligned embeddings with open-domain LMs BERT and RoBERTa, and fine-tune them for two MRC tasks namely span detection (COVID-QA) and multiple-choice questions (PubMedQA). On the COVID-QA dataset, we see that our approach allows these models to perform similar to their domain-specific counterparts, Bio/Sci-BERT, as evidenced by the Exact Match (EM) metric. With regards to PubMedQA, we observe an overall improvement in accuracy while the F1 stays relatively the same over the domain-specific models.

翻译：机器阅读理解（MRC）一直是自然语言处理领域的长期难题，随着基于Transformer架构的BERT系列语言模型的问世，这一问题已取得长足进展。然而，当在通用文本语料上训练的BERT变体应用于特定领域文本时，由于领域迁移（即训练数据与下游应用数据在体裁/主题上的差异）会导致模型性能不可避免地下滑。知识图谱作为开放或封闭领域信息的知识库，先前研究表明其可用于提升通用Transformer在特定领域应用中的表现。基于现有工作，我们提出了一种利用多层感知机（MLP）对齐并整合知识图谱嵌入与预训练语言模型（LM）嵌入空间的方法。我们将对齐后的嵌入与开放域语言模型BERT和RoBERTa融合，并针对两项MRC任务（跨度检测任务COVID-QA与多选题任务PubMedQA）进行微调。在COVID-QA数据集上，我们的方法使模型性能接近其领域特定版本（Bio/Sci-BERT），精确匹配（EM）指标证实了这一点。针对PubMedQA，我们发现模型准确率总体提升，而F1值与领域特定模型基本持平。