We study the probabilistic modeling performed by Autoregressive Large Language Models through the angle of time directionality. We empirically find a time asymmetry exhibited by such models in their ability to model natural language: a difference in the average log-perplexity when trying to predict the next token versus when trying to predict the previous one. This difference is at the same time subtle and very consistent across various modalities (language, model size, training time, ...). Theoretically, this is surprising: from an information-theoretic point of view, there should be no such difference. We provide a theoretical framework to explain how such an asymmetry can appear from sparsity and computational complexity considerations, and outline a number of perspectives opened by our results.

翻译：我们通过时间方向性的视角研究自回归大型语言模型所执行的概率建模。通过实验发现，这类模型在建模自然语言时存在时间不对称性：预测下一个词元与预测前一个词元时，平均对数困惑度存在差异。这种差异既微妙又具有高度一致性（涉及语言、模型规模、训练时长等不同模态）。从理论层面看，这一现象令人惊讶：基于信息论视角，此类差异本不应存在。我们构建了一个理论框架，从稀疏性与计算复杂性的角度阐释这种不对称性如何产生，并概述了研究结果所开启的多重视角。