Despite exceptional capabilities in knowledge-intensive tasks, Large Language Models (LLMs) face a critical gap in understanding how they internalize new knowledge, particularly how to structurally embed acquired knowledge in their neural computations. We address this issue through the lens of knowledge circuit evolution, identifying computational subgraphs that facilitate knowledge storage and processing. Our systematic analysis of circuit evolution throughout continual pre-training reveals several key findings: (1) the acquisition of new knowledge is influenced by its relevance to pre-existing knowledge; (2) the evolution of knowledge circuits exhibits a distinct phase shift from formation to optimization; (3) the evolution of knowledge circuits follows a deep-to-shallow pattern. These insights not only advance our theoretical understanding of the mechanisms of new knowledge acquisition in LLMs, but also provide potential implications for improving continual pre-training strategies to enhance model performance. Code and data will be available at https://github.com/zjunlp/DynamicKnowledgeCircuits.

翻译：尽管大语言模型在知识密集型任务中展现出卓越能力，但学界对其如何内化新知识——特别是如何将习得知识结构化地嵌入神经计算过程——仍存在关键认知缺口。本文通过知识回路演化的视角切入，识别出促进知识存储与处理的计算子图。通过对持续预训练全过程中回路演化的系统分析，我们获得以下关键发现：（1）新知识的获取受其与既有知识相关性的影响；（2）知识回路的演化呈现从形成阶段到优化阶段的显著相变；（3）知识回路的演化遵循由深及浅的模式。这些发现不仅深化了我们对大语言模型新知识获取机制的理论理解，也为改进持续预训练策略以提升模型性能提供了潜在启示。代码与数据将在 https://github.com/zjunlp/DynamicKnowledgeCircuits 公开。