For most natural language processing tasks, the dominant practice is to finetune large pretrained transformer models (e.g., BERT) using smaller downstream datasets. Despite the success of this approach, it remains unclear to what extent these gains are attributable to the massive background corpora employed for pretraining versus to the pretraining objectives themselves. This paper introduces a large-scale study of self-pretraining, where the same (downstream) training data is used for both pretraining and finetuning. In experiments addressing both ELECTRA and RoBERTa models and 10 distinct downstream classification datasets, we observe that self-pretraining rivals standard pretraining on the BookWiki corpus (despite using around $10\times$--$500\times$ less data), outperforming the latter on $7$ and $5$ datasets, respectively. Surprisingly, these task-specific pretrained models often perform well on other tasks, including the GLUE benchmark. Besides classification tasks, self-pretraining also provides benefits on structured output prediction tasks such as span based question answering and commonsense inference, often providing more than $50\%$ of the performance boosts provided by pretraining on the BookWiki corpus. Our results hint that in many scenarios, performance gains attributable to pretraining are driven primarily by the pretraining objective itself and are not always attributable to the use of external pretraining data in massive amounts. These findings are especially relevant in light of concerns about intellectual property and offensive content in web-scale pretraining data.

翻译：对于大多数自然语言处理任务，主流做法是使用较小的下游数据集对大规模预训练Transformer模型（如BERT）进行微调。尽管这一方法取得了成功，但其性能提升在多大程度上归因于预训练所使用的大规模背景语料库，又在多大程度上归因于预训练目标本身，目前仍不清楚。本文介绍了一项关于自预训练的大规模研究，其中相同的（下游）训练数据同时用于预训练和微调。在针对ELECTRA和RoBERTa模型以及10个不同的下游分类数据集进行的实验中，我们观察到自预训练的性能可与在BookWiki语料库上的标准预训练相媲美（尽管使用的数据量减少了约10至500倍），并分别在7个和5个数据集上优于后者。令人惊讶的是，这些任务特定的预训练模型通常在其他任务（包括GLUE基准测试）上表现良好。除分类任务外，自预训练还为结构化输出预测任务（如基于跨度的问题回答和常识推理）带来益处，其性能提升幅度通常超过BookWiki语料库预训练所提供性能提升的50%。我们的研究结果表明，在许多场景中，归因于预训练的性能提升主要源于预训练目标本身，而非总是依赖于大规模外部预训练数据的使用。这些发现对于涉及网络规模预训练数据中的知识产权和不当内容问题尤为重要。