Training large language models (LLMs) is computationally expensive, partly because the loss exhibits slow power-law convergence whose origin remains debatable. Through systematic analysis of toy models and empirical evaluation of LLMs, we show that this behavior can arise intrinsically from the use of softmax and cross-entropy. When learning peaked probability distributions, e.g., next-token distributions, these components yield power-law vanishing losses and gradients, creating a fundamental optimization bottleneck. This ultimately leads to power-law time scaling of the loss with a universal exponent of $1/3$. Our results provide a mechanistic explanation for observed neural scaling and suggest new directions for improving LLM training efficiency.

翻译：大型语言模型（LLM）的训练计算成本高昂，部分原因是其损失函数呈现缓慢的幂律收敛现象，其起源仍存争议。通过对玩具模型的系统分析及对LLM的实证评估，我们发现该行为可能本质源于softmax函数与交叉熵的使用。在学习峰值概率分布（如下一标记分布）时，这些组件会产生幂律衰减的损失与梯度，从而形成根本性的优化瓶颈。这最终导致损失函数呈现幂律时间标度关系，其普适指数为$1/3$。我们的研究结果为观测到的神经标度律提供了机制性解释，并为提升LLM训练效率指出了新方向。