Transformer-based language models have achieved impressive success in various natural language processing tasks due to their ability to capture complex dependencies and contextual information using self-attention mechanisms. However, they are not without limitations. These limitations include hallucinations, where they produce incorrect outputs with high confidence, and alignment issues, where they generate unhelpful and unsafe outputs for human users. These limitations stem from the absence of implicit and missing context in the data alone. To address this, researchers have explored augmenting these models with external knowledge from knowledge graphs to provide the necessary additional context. However, the ad-hoc nature of existing methods makes it difficult to properly analyze the effects of knowledge infusion on the many moving parts or components of a transformer. This paper introduces a systematic method for infusing knowledge into different components of a transformer-based model. A modular framework is proposed to identify specific components within the transformer architecture, such as the self-attention mechanism, encoder layers, or the input embedding layer, where knowledge infusion can be applied. Additionally, extensive experiments are conducted on the General Language Understanding Evaluation (GLUE) benchmark tasks, and the findings are reported. This systematic approach aims to facilitate more principled approaches to incorporating knowledge into language model architectures.

翻译：基于Transformer的语言模型凭借其利用自注意力机制捕获复杂依赖关系与上下文信息的能力，在各类自然语言处理任务中取得了显著成功。然而，这些模型仍存在局限性，包括高置信度输出错误内容的幻觉现象，以及生成对用户无益甚至不安全输出的对齐问题。这些局限源于数据中隐含上下文与缺失语境的缺位。为此，研究者尝试通过注入知识图谱中的外部知识来补充必要上下文以增强模型性能。但现有方法因缺乏系统性，难以准确分析知识注入对Transformer众多可变模块产生的具体影响。本文提出一种系统化方法，可将知识注入至Transformer模型的各个组件中。研究构建了模块化框架，能精确定位Transformer架构中可实施知识注入的具体组件，例如自注意力机制、编码层或输入嵌入层。此外，本文在通用语言理解评估（GLUE）基准任务上开展了大量实验并报告了研究结果。这种系统化方法旨在推动将知识整合到语言模型架构中的更规范化研究路径。