Recent works in Event Argument Extraction (EAE) have focused on improving model generalizability to cater to new events and domains. However, standard benchmarking datasets like ACE and ERE cover less than 40 event types and 25 entity-centric argument roles. Limited diversity and coverage hinder these datasets from adequately evaluating the generalizability of EAE models. In this paper, we first contribute by creating a large and diverse EAE ontology. This ontology is created by transforming FrameNet, a comprehensive semantic role labeling (SRL) dataset for EAE, by exploiting the similarity between these two tasks. Then, exhaustive human expert annotations are collected to build the ontology, concluding with 115 events and 220 argument roles, with a significant portion of roles not being entities. We utilize this ontology to further introduce GENEVA, a diverse generalizability benchmarking dataset comprising four test suites, aimed at evaluating models' ability to handle limited data and unseen event type generalization. We benchmark six EAE models from various families. The results show that owing to non-entity argument roles, even the best-performing model can only achieve 39% F1 score, indicating how GENEVA provides new challenges for generalization in EAE. Overall, our large and diverse EAE ontology can aid in creating more comprehensive future resources, while GENEVA is a challenging benchmarking dataset encouraging further research for improving generalizability in EAE. The code and data can be found at https://github.com/PlusLabNLP/GENEVA.

翻译：[近期事件论元抽取（EAE）研究聚焦于提升模型泛化能力以应对新事件和领域。然而，ACE和ERE等标准基准数据集仅覆盖不到40种事件类型和25种以实体为中心的论元角色。有限的多样性和覆盖面导致这些数据集难以充分评估EAE模型的泛化能力。本文首先通过构建大规模、多样化的EAE本体作出贡献。该本体利用事件抽取与语义角色标注（SRL）任务间的相似性，将综合性SRL数据集FrameNet转换为EAE格式。随后通过详尽的专家人工标注完成本体构建，最终包含115种事件和220种论元角色，其中显著部分为非实体角色。基于此本体，我们进一步提出GENEVA——一个包含四个测试套件的多样化泛化能力基准数据集，旨在评估模型处理有限数据和未见事件类型泛化的能力。我们对来自不同家族的六个EAE模型进行了基准测试。结果表明，由于非实体论元角色的存在，即使性能最优的模型也仅能达到39%的F1分数，揭示了GENEVA为EAE泛化研究带来的新挑战。总体而言，我们所构建的大规模多样化EAE本体有助于创建更全面的未来资源，而GENEVA作为具有挑战性的基准数据集，将激励提升EAE泛化能力的进一步研究。代码与数据详见https://github.com/PlusLabNLP/GENEVA。]