Current causal text mining datasets vary in objectives, data coverage, and annotation schemes. These inconsistent efforts prevent modeling capabilities and fair comparisons of model performance. Furthermore, few datasets include cause-effect span annotations, which are needed for end-to-end causal relation extraction. To address these issues, we propose UniCausal, a unified benchmark for causal text mining across three tasks: (I) Causal Sequence Classification, (II) Cause-Effect Span Detection and (III) Causal Pair Classification. We consolidated and aligned annotations of six high quality, mainly human-annotated, corpora, resulting in a total of 58,720, 12,144 and 69,165 examples for each task respectively. Since the definition of causality can be subjective, our framework was designed to allow researchers to work on some or all datasets and tasks. To create an initial benchmark, we fine-tuned BERT pre-trained language models to each task, achieving 70.10% Binary F1, 52.42% Macro F1, and 84.68% Binary F1 scores respectively.

翻译：当前因果文本挖掘数据集在目标、数据覆盖范围及标注方案上存在差异。这些不一致的努力阻碍了建模能力的发展及模型性能的公平比较。此外，仅有少数数据集包含因果跨度标注，而这是端到端因果关系抽取所必需的。为解决这些问题，我们提出UniCausal——一个针对三项任务的统一因果文本挖掘基准：（I）因果序列分类，（II）因果跨度检测，（III）因果对分类。我们整合并对齐了六个高质量（主要为人工标注）语料库的标注，分别为每项任务生成了58,720、12,144和69,165个示例。由于因果性定义可能具有主观性，我们的框架设计允许研究人员选择部分或全部数据集与任务进行工作。为建立初始基准，我们微调了BERT预训练语言模型以适配每项任务，分别取得了70.10%的二元F1值、52.42%的宏F1值及84.68%的二元F1值。