On a variety of tasks, the performance of neural networks predictably improves with training time, dataset size and model size across many orders of magnitude. This phenomenon is known as a neural scaling law. Of fundamental importance is the compute-optimal scaling law, which reports the performance as a function of units of compute when choosing model sizes optimally. We analyze a random feature model trained with gradient descent as a solvable model of network training and generalization. This reproduces many observations about neural scaling laws. First, our model makes a prediction about why the scaling of performance with training time and with model size have different power law exponents. Consequently, the theory predicts an asymmetric compute-optimal scaling rule where the number of training steps are increased faster than model parameters, consistent with recent empirical observations. Second, it has been observed that early in training, networks converge to their infinite-width dynamics at a rate $1/\textit{width}$ but at late time exhibit a rate $\textit{width}^{-c}$, where $c$ depends on the structure of the architecture and task. We show that our model exhibits this behavior. Lastly, our theory shows how the gap between training and test loss can gradually build up over time due to repeated reuse of data.

翻译：在各种任务中，神经网络性能随训练时间、数据集规模和模型规模的提升呈现可预测的改进，跨越多个数量级。这一现象被称为神经缩放定律。其中，计算最优缩放定律至关重要，它报告了在最优选择模型规模时，性能作为计算单位函数的表现。我们分析了采用梯度下降训练的随机特征模型，将其作为网络训练与泛化的可解模型。该模型复现了神经缩放定律的多个观测结果。首先，我们的模型对为何训练时间与模型规模对性能的缩放遵循不同幂律指数进行了预测。由此，该理论预言了一种非对称的计算最优缩放规则：训练步数的增加速度应快于模型参数，这与近期经验观测结果一致。其次，已有研究发现，在训练早期，网络会以速率 $1/\textit{width}$ 收敛至其无穷宽度动力学，而后期则以速率 $\textit{width}^{-c}$ 收敛，其中 $c$ 取决于架构与任务的结构。我们证明模型展现出此类行为。最后，我们的理论揭示了训练损失与测试损失之间的差距如何因数据的重复使用而随时间逐渐累积。