We inspect the deductive connection between the neural scaling law and Zipf's law -- two statements discussed in machine learning and quantitative linguistics. The neural scaling law describes how the cross entropy rate of a foundation model -- such as a large language model -- changes with respect to the amount of training tokens, parameters, and compute. By contrast, Zipf's law posits that the distribution of tokens exhibits a power law tail. Whereas similar claims have been made in more specific settings, we show that the neural scaling law is a consequence of Zipf's law under certain broad assumptions that we reveal systematically. The derivation steps are as follows: We derive Heaps' law on the vocabulary growth from Zipf's law, Hilberg's hypothesis on the entropy scaling from Heaps' law, and the neural scaling from Hilberg's hypothesis. We illustrate these inference steps by a toy example of the Santa Fe process that satisfies all the four statistical laws.

翻译：本文考察了神经缩放定律与齐普夫定律之间的演绎关联——这两种表述分别常见于机器学习与计量语言学领域。神经缩放定律描述了基础模型（如大语言模型）的交叉熵率如何随训练词元数量、参数量及计算量的变化而变化。相比之下，齐普夫定律则主张词元分布呈现幂律尾特征。尽管已有研究在更具体的设定中提出过类似主张，我们通过系统化揭示的若干广义假设，证明了齐普夫定律可推导出神经缩放定律。推导过程包含以下步骤：首先从齐普夫定律导出描述词汇增长的赫普斯定律，继而从赫普斯定律导出描述熵缩放的希尔伯格假说，最终从希尔伯格假说导出神经缩放定律。我们通过满足全部四条统计定律的圣塔菲过程玩具示例，对这些推导步骤进行了具体阐释。