Language models trained on observed sequences are often described as learning the conditional distribution of the next token given previous tokens. This description is only conditionally correct. A model trained on realized token trajectories does not observe full conditional laws; it receives sampled continuations. Moreover, real language generation is conditioned not only on previous words but also on non-textual circumstances: facts, events, intentions, goals, beliefs, social context, and task-specific constraints. This paper distinguishes three objects that are often conflated: the full conditional language process conditioned on latent circumstances, the marginal text-only process obtained by integrating those circumstances out, and the model-induced distribution learned from finite observed corpora. The paper argues that interpreting model training as estimating the marginal text-only law requires strong assumptions of stationarity, representativeness, and ergodicity, assumptions that are standard in statistical estimation but problematic when applied to heterogeneous language corpora. Even if these assumptions hold, the marginal text-only law is useful only when the observed prefix is an approximately sufficient statistic for the latent circumstances relevant to continuation. In information-theoretic terms, usefulness requires that the residual conditional mutual information between the next token and the omitted circumstances, given the observed text, be small. The paper then extends this argument to heterogeneous training corpora. Finally, the paper interprets Retrieval Augmented Generation (RAG) and tool use as conditional sufficiency devices.

翻译：在观测序列上训练的语言模型通常被描述为学习给定前序词条件下的下一个词的条件分布。这一描述仅在某些条件下成立。基于已实现的词元轨迹训练的模型并未观察到完整的条件分布律，它仅接收采样得到的延续序列。此外，真实的语言生成不仅受前序词语的制约，还受到非文本性环境因素的影响：事实、事件、意图、目标、信念、社会语境以及任务特定约束。本文区分了三个常被混淆的对象：以潜在环境因素为条件的全条件语言过程、通过积分消去环境因素得到的纯文本边际过程、以及从有限观测语料库中学到的模型诱导分布。本文论证，将模型训练解释为对纯文本边际分布律的估计需要极强的平稳性、代表性和遍历性假设，这些假设在统计估计中是标准要求，但在应用于异质性语言语料库时存在问题。即使这些假设成立，纯文本边际分布律仅在观测前缀是关于影响续接的相关潜在环境因素的近似充分统计量时才有用。从信息论角度而言，有用性要求给定观测文本后下一个词与被忽略环境因素之间的残差条件互信息很小。本文随后将此论证推广到异质性训练语料库。最后，本文将检索增强生成（RAG）和工具使用解释为条件充分性机制。