Large Language Models (LLMs) trained on web-scale text corpora have been shown to capture world knowledge in their parameters. However, the mechanism by which language models store different types of knowledge is poorly understood. In this work, we examine two types of knowledge relating to temporally sensitive entities and demonstrate that each type is localized to different sets of parameters within the LLMs. We hypothesize that the lack of consideration of the locality of knowledge in existing continual learning methods contributes to both: the failed uptake of new information, and catastrophic forgetting of previously learned information. We observe that sequences containing references to updated and newly mentioned entities exhibit larger gradient norms in a subset of layers. We demonstrate that targeting parameter updates to these relevant layers can improve the performance of continually pretraining on language containing temporal drift.

翻译：基于网络规模文本语料库训练的大型语言模型（LLM）已被证明能够在其参数中捕获世界知识。然而，语言模型存储不同类型知识的机制尚不明确。本研究针对时间敏感实体的两类知识展开分析，证明每种知识类型在LLM内部定位于不同的参数集合。我们推测，现有持续学习方法对知识局部性的忽视导致了双重问题：新信息吸收失败与已学信息的灾难性遗忘。通过实验观察到，涉及更新实体与新提及实体的文本序列在特定层中呈现更大的梯度范数。研究证明，针对这些相关层进行参数更新能够有效提升包含时序漂移的语言数据持续预训练性能。