Pretrained language models (LMs) encode implicit representations of knowledge in their parameters. However, localizing these representations and disentangling them from each other remains an open problem. In this work, we investigate whether pretrained language models contain various knowledge-critical subnetworks: particular sparse computational subgraphs responsible for encoding specific knowledge the model has memorized. We propose a multi-objective differentiable weight masking scheme to discover these subnetworks and show that we can use them to precisely remove specific knowledge from models while minimizing adverse effects on the behavior of the original language model. We demonstrate our method on multiple GPT2 variants, uncovering highly sparse subnetworks (98%+) that are solely responsible for specific collections of relational knowledge. When these subnetworks are removed, the remaining network maintains most of its initial capacity (modeling language and other memorized relational knowledge) but struggles to express the removed knowledge, and suffers performance drops on examples needing this removed knowledge on downstream tasks after finetuning.

翻译：预训练语言模型在其参数中编码了隐含的知识表征。然而，定位这些表征并将其彼此分离仍是一个未解难题。本研究探究预训练语言模型是否包含多种知识关键子网络：即负责编码模型所记忆的特定知识的稀疏计算子图。我们提出一种多目标可微分权重掩蔽方案来发现这些子网络，并证明我们可以利用它们精确地从模型中移除特定知识，同时将对原始语言模型行为的不利影响降至最低。我们在多种GPT2变体上验证了该方法，发现了高度稀疏（98%以上）且仅负责特定关系知识集合的子网络。当这些子网络被移除后，剩余网络虽能维持大部分初始能力（建模语言及其他记忆的关系知识），但难以表达被移除的知识，并在微调后需要这些被移除知识的下游任务示例上出现性能下降。