Surprisal theory links human processing effort to the predictability of an upcoming linguistic unit, but empirical work often leaves the notion of a unit underspecified. In practice, experimental stimuli are segmented into linguistically motivated units (e.g., words), while pretrained language models assign probability mass to a fixed token alphabet that typically does not align with those units. As a result, surprisal-based predictors depend implicitly on ad hoc procedures that conflate two distinct modeling choices: the definition of the unit of analysis and the choice of regions of interest over which predictions are evaluated. In this paper, we disentangle these choices and give a unified framework for reasoning about surprisal over arbitrary unit inventories. We argue that surprisal-based analyses should make these choices explicit and treat tokenization as an implementation detail rather than a scientific primitive.

翻译：惊奇理论将人类加工处理难度与即将出现的语言单位的可预测性联系起来，但实证研究往往未对"单位"这一概念进行充分限定。在实践中，实验刺激被分割成具有语言学意义的单位（如词语），而预训练语言模型将概率质量分配给通常与这些单位不一致的固定词元表。因此，基于惊奇的预测指标隐含地依赖于将两种不同的建模选择混为一谈的临时性程序：分析单位的定义以及用于评估预测的感兴趣区域的选择。本文对这些选择进行了区分，并提出了一个统一框架来推理任意单位清单上的惊奇。我们认为，基于惊奇的分析应明确这些选择，并将词元化视为实现细节而非科学基元。