We study the probabilistic modeling performed by Autoregressive Large Language Models (LLMs) through the angle of time directionality, addressing a question first raised in (Shannon, 1951). For large enough models, we empirically find a time asymmetry in their ability to learn natural language: a difference in the average log-perplexity when trying to predict the next token versus when trying to predict the previous one. This difference is at the same time subtle and very consistent across various modalities (language, model size, training time, ...). Theoretically, this is surprising: from an information-theoretic point of view, there should be no such difference. We provide a theoretical framework to explain how such an asymmetry can appear from sparsity and computational complexity considerations, and outline a number of perspectives opened by our results.

翻译：本研究从时间方向性的角度探讨自回归大语言模型（LLMs）的概率建模特性，回应了（Shannon, 1951）首次提出的问题。对于足够大规模的模型，我们通过实证发现其学习自然语言的能力存在时间不对称性：模型在预测下一词元与预测上一词元时的平均对数困惑度存在差异。这种差异虽细微，却在多种模态（语言类型、模型规模、训练时长等）中表现出高度一致性。从理论角度看，这一发现令人惊讶：根据信息论观点，此类差异不应存在。我们提出了一个理论框架，从稀疏性与计算复杂度的角度解释这种不对称性的产生机制，并阐述了本研究结果所开启的若干研究方向。