Large language models leverage not only parametric knowledge acquired during training but also in-context knowledge provided at inference time, despite the absence of explicit training objectives for using both sources. Prior work has further shown that when these knowledge sources conflict, models resolve the tension based on their internal confidence, preferring parametric knowledge for high-confidence facts while deferring to contextual information for less familiar ones. However, the training conditions that give rise to such knowledge utilization behaviors remain unclear. To address this gap, we conduct controlled experiments in which we train language models while systematically manipulating key properties of the training data. Our results reveal a counterintuitive finding: three properties commonly regarded as detrimental must co-occur for robust knowledge utilization and conflict resolution to emerge: (i) intra-document repetition of information, (ii) a moderate degree of within-document inconsistency, and (iii) a skewed knowledge frequency distribution. We further validate that the same training dynamics observed in our controlled setting also arise during real-world language model pretraining, and we analyze how post-training procedures can reshape models' knowledge preferences. Together, our findings provide concrete empirical guidance for training language models that harmoniously integrate parametric and in-context knowledge.

翻译：大型语言模型不仅利用训练期间获得的参数化知识，还利用推理时提供的上下文知识，尽管缺乏明确训练目标来同时使用这两种知识源。先前研究进一步表明，当这些知识源发生冲突时，模型会根据其内部置信度来化解矛盾：对高置信度事实倾向于采用参数化知识，而对较不熟悉的事实则倾向于采纳上下文信息。然而，导致此类知识利用行为的训练条件仍不明确。为填补这一空白，我们进行了受控实验，在训练语言模型时系统性地操纵训练数据的关键属性。我们的研究结果揭示了一个反直觉的发现：三种通常被认为有害的属性必须同时出现，才能形成稳健的知识利用与冲突化解能力：（i）信息在文档内的重复出现，（ii）适度的文档内部不一致性，以及（iii）倾斜的知识频率分布。我们进一步验证了在受控环境中观察到的训练动态同样出现在现实世界的语言模型预训练过程中，并分析了后训练程序如何重塑模型的知识偏好。综合而言，我们的研究结果为训练能够和谐整合参数化知识与上下文知识的语言模型提供了具体的实证指导。