Natural Language Inference (NLI) tasks involving temporal inference remain challenging for pre-trained language models (LMs). Although various datasets have been created for this task, they primarily focus on English and do not address the need for resources in other languages. It is unclear whether current LMs realize the generalization capacity for temporal inference across languages. In this paper, we present Jamp, a Japanese NLI benchmark focused on temporal inference. Our dataset includes a range of temporal inference patterns, which enables us to conduct fine-grained analysis. To begin the data annotation process, we create diverse inference templates based on the formal semantics test suites. We then automatically generate diverse NLI examples by using the Japanese case frame dictionary and well-designed templates while controlling the distribution of inference patterns and gold labels. We evaluate the generalization capacities of monolingual/multilingual LMs by splitting our dataset based on tense fragments (i.e., temporal inference patterns). Our findings demonstrate that LMs struggle with specific linguistic phenomena, such as habituality, indicating that there is potential for the development of more effective NLI models across languages.

翻译：摘要：涉及时序推理的自然语言推理任务对预训练语言模型仍具有挑战性。尽管已有多种数据集为此任务创建，但它们主要针对英语，未能满足其他语言的需求。目前尚不清楚现有语言模型是否具备跨语言时序推理的泛化能力。本文提出Jamp，一个专注于时序推理的日语自然语言推理基准数据集。我们的数据集包含多种时序推理模式，可支持细粒度分析。为启动数据标注流程，我们基于形式语义测试套件创建多样化推理模板。随后利用日语格框架词典与精心设计的模板，在控制推理模式与真实标签分布的同时自动生成多样化自然语言推理实例。通过基于时态片段（即时序推理模式）划分数据集，我们评估了单语言/多语言语言模型的泛化能力。实验结果表明，语言模型在处理特定语言现象（如习惯性）时存在困难，这为跨语言更高效的自然语言推理模型开发提供了改进空间。