Computational narrative understanding studies the identification, description, and interaction of the elements of a narrative: characters, attributes, events, and relations. Narrative research has given considerable attention to defining and classifying character types. However, these character-type taxonomies do not generalize well because they are small, too simple, or specific to a domain. We require robust and reliable benchmarks to test whether narrative models truly understand the nuances of the character's development in the story. Our work addresses this by curating the Chatter dataset that labels whether a character portrays some attribute for 88148 character-attribute pairs, encompassing 2998 characters, 13324 attributes and 660 movies. We validate a subset of Chatter, called ChatterEval, using human annotations to serve as an evaluation benchmark for the character attribution task in movie scripts. ChatterEval assesses narrative understanding and the long-context modeling capacity of language models.

翻译：计算叙事理解研究叙事元素（角色、属性、事件及关系）的识别、描述与交互。叙事研究对角色类型的定义与分类已给予相当关注，然而现有角色类型分类体系因规模有限、过于简化或领域特定而泛化能力不足。我们需要稳健可靠的基准来检验叙事模型是否真正理解故事中角色发展的细微演变。本研究通过构建Chatter数据集应对这一挑战，该数据集对88148个角色-属性对（涵盖2998个角色、13324种属性及660部电影）标注了角色是否呈现特定属性。我们通过人工标注验证了Chatter的子集ChatterEval，将其作为电影剧本中角色属性任务的评估基准。ChatterEval可用于评估语言模型的叙事理解能力与长上下文建模能力。