Incremental models that process sentences one token at a time will sometimes encounter points where more than one interpretation is possible. Causal models are forced to output one interpretation and continue, whereas models that can revise may edit their previous output as the ambiguity is resolved. In this work, we look at how restart-incremental Transformers build and update internal states, in an effort to shed light on what processes cause revisions not viable in autoregressive models. We propose an interpretable way to analyse the incremental states, showing that their sequential structure encodes information on the garden path effect and its resolution. Our method brings insights on various bidirectional encoders for contextualised meaning representation and dependency parsing, contributing to show their advantage over causal models when it comes to revisions.

翻译：增量模型逐词处理句子时，有时会遇到存在多种可能解释的节点。因果模型被迫输出一种解释并继续推进，而具备修订能力的模型可在歧义消解时修正先前的输出。本研究通过考察重启增量式Transformer如何构建与更新内部状态，试图揭示哪些处理过程导致了自回归模型无法实现的修订行为。我们提出一种可解释的方法来分析增量状态，证明其序列结构编码了花园路径效应及其消解的信息。该方法为多种用于语境化意义表示和依存句法分析的双向编码器提供了新见解，有助于展示其在处理修订任务时相较于因果模型的优势。