The events in a narrative are understood as a coherent whole via the underlying states of their participants. Often, these participant states are not explicitly mentioned, instead left to be inferred by the reader. A model that understands narratives should likewise infer these implicit states, and even reason about the impact of changes to these states on the narrative. To facilitate this goal, we introduce a new crowdsourced English-language, Participant States dataset, PASTA. This dataset contains inferable participant states; a counterfactual perturbation to each state; and the changes to the story that would be necessary if the counterfactual were true. We introduce three state-based reasoning tasks that test for the ability to infer when a state is entailed by a story, to revise a story conditioned on a counterfactual state, and to explain the most likely state change given a revised story. Experiments show that today's LLMs can reason about states to some degree, but there is large room for improvement, especially in problems requiring access and ability to reason with diverse types of knowledge (e.g. physical, numerical, factual).
翻译:叙事中的事件通过其参与者的潜在状态被理解为连贯的整体。通常,这些参与者状态并未被明确提及,而是需要读者自行推断。理解叙事的模型应同样推断这些隐含状态,甚至推理状态变化对叙事的影响。为促进这一目标,我们引入了一个新的大规模众包英语数据集——参与者状态数据集PASTA。该数据集包含可推断的参与者状态;每个状态的反事实扰动;以及若反事实成立时故事所需的变化。我们提出三项基于状态的推理任务,测试模型能否推断故事所蕴含的状态、基于反事实状态修订故事,以及解释给定修订故事后最可能的状态变化。实验表明,当前的大语言模型(LLMs)能在一定程度上推理状态,但仍有较大改进空间,尤其在需要获取并推理多种类型知识(如物理、数值、事实知识)的问题上。