Language model based pre-trained models such as BERT have provided significant gains across different NLP tasks. In this paper, we study different types of pre-trained transformer based models such as auto-regressive models (GPT-2), auto-encoder models (BERT), and seq2seq models (BART) for conditional data augmentation. We show that prepending the class labels to text sequences provides a simple yet effective way to condition the pre-trained models for data augmentation. On three classification benchmarks, pre-trained Seq2Seq model outperforms other models. Further, we explore how different pre-trained model based data augmentation differs in-terms of data diversity, and how well such methods preserve the class-label information.

翻译：基于语言的预先培训模型,如BERT,在不同的国家劳工计划任务中取得了显著成果。在本文中,我们研究了不同类型的预先培训的变压器模型,如自动递减模型(GPT-2)、自动编码模型(BERT)和有条件数据扩增的后继2seq模型(BART)等。我们表明,在文本序列中预留类标签为确定预先培训的数据扩增模型提供了简单而有效的条件。在三个分类基准方面,预先培训的Seq2Seq模型优于其他模型。此外,我们探索了不同的预先培训的模型数据扩增在数据多样性的时期中如何不同,以及这类方法如何保护类别标签信息。