Large language models (LLMs) such as GPT-4 are very powerful and can process different kinds of natural language processing (NLP) tasks. However, it can be difficult to interpret the results due to the multi-layer nonlinear model structure and millions of parameters. A lack of clarity and understanding of how the language models (LMs) work can make them unreliable, difficult to trust, and potentially dangerous for use in real-world scenarios. Most recent works exploit attention weights to provide explanations for LM predictions. However, pure attention-based explanations are unable to support the growing complexity of LMs, and cannot reason about their decision-making processes. We propose LMExplainer, a knowledge-enhanced explainer for LMs that can provide human-understandable explanations. We use a knowledge graph (KG) and a graph attention neural network to extract the key decision signals of the LM. We further explore whether interpretation can also help the AI understand the task better. Our experimental results show that LMExplainer outperforms existing LM+KG methods on CommonsenseQA and OpenBookQA. We compare the explanation results with generated explanation methods and human-annotated results. The comparison shows our method can provide more comprehensive and clearer explanations. LMExplainer demonstrates the potential to enhance model performance and furnish explanations for the LM reasoning process in natural language.

翻译：大型语言模型（如GPT-4）功能强大，可处理多种自然语言处理任务。然而，由于其多层非线性模型结构和数百万参数，结果难以解释。对语言模型工作原理缺乏清晰认知可能导致其不可靠、难以信任，并在实际应用场景中具有潜在风险。近期研究主要利用注意力权重为语言模型预测提供解释，但纯注意力机制的解释无法支撑日益复杂的语言模型，也无法推理其决策过程。我们提出LMExplainer——一种面向语言模型的知识增强解释器，可提供人类可理解的解释。该方法通过知识图谱与图注意力神经网络提取语言模型关键决策信号，并进一步探究解释能否帮助人工智能更好地理解任务。实验表明，LMExplainer在CommonsenseQA和OpenBookQA数据集上优于现有语言模型+知识图谱方法。通过将解释结果与生成式解释方法及人工标注结果对比，该方法的解释更全面清晰。LMExplainer展现了提升模型性能、用自然语言解释语言模型推理过程的潜力。