Keeping track of how states and relations of entities change as a text or dialog unfolds is a key prerequisite to discourse understanding. Despite this fact, there have been few systematic investigations into the ability of large language models (LLMs) to track discourse entities. In this work, we present a task to probe to what extent a language model can infer the final state of an entity given an English description of the initial state and a series of state-changing operations. We use this task to first investigate whether Flan-T5, GPT-3 and GPT-3.5 can track the state of entities, and find that only GPT-3.5 models, which have been pretrained on large amounts of code, exhibit this ability. We then investigate whether smaller models pretrained primarily on text can learn to track entities, through finetuning T5 on several training/evaluation splits. While performance degrades for more complex splits, we find that even for splits with almost no lexical overlap between training and evaluation, a finetuned model can often perform non-trivial entity tracking. Taken together, these results suggest that language models can learn to track entities but pretraining on large text corpora alone does not make this capacity surface.

翻译：随着文本或对话的展开，追踪实体状态与关系的变化是语篇理解的关键前提。然而，针对大型语言模型（LLM）追踪语篇实体能力的系统性研究仍较为缺乏。本文提出一项任务，旨在探究语言模型在给定初始状态的英文描述及一系列状态变更操作后，推断实体最终状态的能力。我们利用该任务首先考察Flan-T5、GPT-3和GPT-3.5是否能够追踪实体状态，发现仅经过大量代码预训练的GPT-3.5模型展现出该能力。随后，我们探究主要在文本上预训练的小型模型能否通过学习追踪实体，具体方法是在多个训练/评估数据划分上对T5进行微调。尽管在更复杂的数据划分中模型性能有所下降，但研究发现即使训练集与评估集几乎不存在词汇重叠，微调模型仍能有效完成非平凡的实体追踪任务。综合而言，这些结果表明语言模型能够学习实体追踪，但仅依靠大规模文本语料预训练尚不足以激发该能力。