Large language models accumulate vast knowledge during pre-training, yet the dynamics governing this acquisition remain poorly understood. This work investigates the learning dynamics of language models on a synthetic factual recall task, uncovering three key findings: First, language models learn in three phases, exhibiting a performance plateau before acquiring precise factual knowledge. Mechanistically, this plateau coincides with the formation of attention-based circuits that support recall. Second, the training data distribution significantly impacts learning dynamics, as imbalanced distributions lead to shorter plateaus. Finally, hallucinations emerge simultaneously with knowledge, and integrating new knowledge into the model through fine-tuning is challenging, as it quickly corrupts its existing parametric memories. Our results emphasize the importance of data distribution in knowledge acquisition and suggest novel data scheduling strategies to accelerate neural network training.

翻译：大型语言模型在预训练过程中积累了海量知识，然而支配这种知识获取的动态机制仍鲜为人知。本研究通过合成事实回忆任务探究语言模型的学习动态，揭示了三个关键发现：首先，语言模型的学习呈现三阶段特征，在掌握精确事实知识前会经历性能平台期。从机制上看，该平台期与支持信息回忆的注意力机制回路的形成同步发生。其次，训练数据分布对学习动态具有显著影响，不平衡的数据分布会导致平台期缩短。最后，幻觉现象与知识获取同步产生，且通过微调将新知识整合到模型中具有挑战性——这会迅速破坏模型已有的参数化记忆。我们的研究结果强调了数据分布在知识获取中的重要性，并为加速神经网络训练提出了创新的数据调度策略。