Language models (LMs), like other neural networks, often favor shortcut heuristics based on surface-level patterns. Although LMs behave like n-gram models early in training, they must eventually learn hierarchical syntactic representations to correctly apply grammatical rules out-of-distribution (OOD). In this work, we use case studies of English grammar to explore how complex, diverse training data drives models to generalize OOD. We construct a framework that unifies our understanding of random variation with training dynamics, rule selection with memorization, and data diversity with complexity. We show that these factors are nuanced, and that intermediate levels of diversity and complexity lead to inconsistent behavior across random seeds and to unstable training dynamics. Our findings emphasize the critical role of training data in shaping generalization patterns and illuminate how competing model strategies lead to inconsistent generalization outcomes across random seeds. Code is available at https://github.com/sunnytqin/concept_comp.git.

翻译：语言模型（Language Models, LMs）与其他神经网络类似，常常偏好基于表层模式的捷径启发式方法。尽管语言模型在训练初期表现得像 n-gram 模型，但它们最终必须学习层次化的句法表示，以在分布外（out-of-distribution, OOD）场景中正确应用语法规则。在本工作中，我们通过英语语法的案例研究，探讨了复杂多样的训练数据如何驱动模型实现 OOD 泛化。我们构建了一个统一框架，将随机变异与训练动态、规则选择与记忆、数据多样性与复杂性纳入统一理解。研究表明，这些因素具有细微差别，中等水平的多样性与复杂性会导致不同随机种子下的行为不一致以及训练动态的不稳定。我们的发现强调了训练数据在塑造泛化模式中的关键作用，并阐明了相互竞争的模型策略如何导致不同随机种子下泛化结果的不一致。代码可在 https://github.com/sunnytqin/concept_comp.git 获取。