It was observed that large language models exhibit a power-law decay of cross entropy with respect to the number of parameters and training tokens. When extrapolated literally, this decay implies that the entropy rate of natural language is zero. To understand this phenomenon -- or an artifact -- better, we construct a simple stationary stochastic process and its memory-based predictor that exhibit a power-law decay of cross entropy with the vanishing entropy rate. Our example is based on previously discussed Santa Fe processes, which decompose a random text into a process of narration and time-independent knowledge. Previous discussions assumed that narration is a memoryless source with Zipf's distribution. In this paper, we propose a model of narration that has the vanishing entropy rate and applies a randomly chosen deterministic sequence called a multiperiodic sequence. Under a suitable parameterization, multiperiodic sequences exhibit asymptotic relative frequencies given by Zipf's law. Remaining agnostic about the value of the entropy rate of natural language, we discuss relevance of similar constructions for language modeling.

翻译：大型语言模型的交叉熵相对于参数数量和训练 token 数量呈现出幂律衰减。若按字面意义外推，该衰减意味着自然语言的熵率为零。为更好地理解这一现象（或假象），我们构建了一个平稳随机过程及其基于记忆的预测器，该模型在熵率趋于零时仍能呈现交叉熵的幂律衰减。我们的示例基于先前讨论的圣达菲过程，该过程将随机文本分解为叙事过程与时间无关知识。以往讨论假设叙事是无记忆源且服从齐普夫分布。本文提出一种具有零熵率的叙事模型，该模型应用称为多周期序列的随机确定性序列。在适当参数化下，多周期序列的渐近相对频率呈现齐普夫定律。在保持对自然语言熵率取值不可知论立场的前提下，我们讨论了类似构造对语言建模的相关性。