We present Pre-trained Machine Reader (PMR), a novel method to retrofit Pre-trained Language Models (PLMs) into Machine Reading Comprehension (MRC) models without acquiring labeled data. PMR is capable of resolving the discrepancy between model pre-training and downstream fine-tuning of existing PLMs, and provides a unified solver for tackling various extraction tasks. To achieve this, we construct a large volume of general-purpose and high-quality MRC-style training data with the help of Wikipedia hyperlinks and design a Wiki Anchor Extraction task to guide the MRC-style pre-training process. Although conceptually simple, PMR is particularly effective in solving extraction tasks including Extractive Question Answering and Named Entity Recognition, where it shows tremendous improvements over previous approaches especially under low-resource settings. Moreover, viewing sequence classification task as a special case of extraction task in our MRC formulation, PMR is even capable to extract high-quality rationales to explain the classification process, providing more explainability of the predictions.

翻译：为实现这一目标,我们在维基百科超链接的帮助下,制作了大量通用和高质量的MRC型培训数据,并设计了一个Wiki Anchor Explication任务,以指导MRC式的培训前进程。虽然在概念上很简单,但PMR在解决包括采掘业问题回答和实体识别在内的抽取任务方面特别有效,这显示了以往方法,特别是在资源贫乏的情况下,在这方面取得了巨大进步。此外,将序列分类任务视为我们MRC的提炼任务的特殊案例,PMR甚至能够提取出高质量的理由解释分类过程,为预测提供更清楚的解释。