Pre-trained language models achieves high performance on machine reading comprehension (MRC) tasks but the results are hard to explain. An appealing approach to make models explainable is to provide rationales for its decision. To facilitate supervised learning of human rationales, here we present PALRACE (Pruned And Labeled RACE), a new MRC dataset with human labeled rationales for 800 passages selected from the RACE dataset. We further classified the question to each passage into 6 types. Each passage was read by at least 26 participants, who labeled their rationales to answer the question. Besides, we conducted a rationale evaluation session in which participants were asked to answering the question solely based on labeled rationales, confirming that the labeled rationales were of high quality and can sufficiently support question answering.

翻译：预先培训的语言模型在机器阅读理解任务中取得了很高的成绩,但结果很难解释。 使模型具有解释性的一个吸引人的方法是为其决定提供理由。为了便利在监督下学习人的理由,我们在此介绍一个带有从机器阅读理解(MRC)数据集中选取800个段落的人类标签理由的新的MRC数据集。我们进一步将问题分为6个段落。每个段落至少有26名与会者阅读,他们用标签说明理由来回答问题。此外,我们举行了一次理由评估会议,邀请与会者仅根据标签理由回答问题,确认标签理由是高质量的,可以充分支持回答问题。