Deep neural networks learn increasingly complex functions over the course of training. Here, we show both empirically and theoretically that learning of the target function is preceded by an early phase in which networks learn the optimal constant solution (OCS) - that is, initial model responses mirror the distribution of target labels, while entirely ignoring information provided in the input. Using a hierarchical category learning task, we derive exact solutions for learning dynamics in deep linear networks trained with bias terms. Even when initialized to zero, this simple architectural feature induces substantial changes in early dynamics. We identify hallmarks of this early OCS phase and illustrate how these signatures are observed in deep linear networks and larger, more complex (and nonlinear) convolutional neural networks solving a hierarchical learning task based on MNIST and CIFAR10. We explain these observations by proving that deep linear networks necessarily learn the OCS during early learning. To further probe the generality of our results, we train human learners over the course of three days on the category learning task. We then identify qualitative signatures of this early OCS phase in terms of the dynamics of true negative (correct-rejection) rates. Surprisingly, we find the same early reliance on the OCS in the behaviour of human learners. Finally, we show that learning of the OCS can emerge even in the absence of bias terms and is equivalently driven by generic correlations in the input data. Overall, our work suggests the OCS as a universal learning principle in supervised, error-corrective learning, and the mechanistic reasons for its prevalence.

翻译：深度神经网络在训练过程中学习日益复杂的函数。本文通过实证与理论分析表明，在目标函数学习阶段之前存在一个早期阶段，网络会首先学习最优常数解（OCS）——即初始模型响应仅反映目标标签的分布特征，而完全忽略输入信息。通过构建层次化类别学习任务，我们推导出带偏置项的深度线性网络学习动态的精确解。即使网络初始化为零权重，这一简单的架构特征仍会显著改变早期学习动态。我们识别了早期OCS阶段的典型特征，并展示了这些特征在深度线性网络以及更复杂（非线性）的卷积神经网络中的表现，这些网络使用基于MNIST和CIFAR10构建的层次学习任务。通过证明深度线性网络在早期学习中必然习得OCS，我们解释了这些现象。为探究研究结果的普适性，我们对人类受试者进行了为期三天的类别学习任务训练，随后根据真阴性率（正确拒绝率）的动态变化识别出早期OCS阶段的定性特征。令人惊讶的是，人类学习者的行为同样表现出对OCS的早期依赖。最后，我们证明即使在没有偏置项的情况下，OCS学习仍可能通过输入数据中的通用相关性驱动而出现。总体而言，我们的研究表明OCS是监督式误差校正学习中普遍存在的学习原则，并揭示了其普遍存在的机制性原因。