In recent years, language models (LMs) have made remarkable progress in advancing the field of natural language processing (NLP). However, the impact of data augmentation (DA) techniques on the fine-tuning (FT) performance of these LMs has been a topic of ongoing debate. In this study, we evaluate the effectiveness of three different FT methods in conjugation with back-translation across an array of 7 diverse NLP tasks, including classification and regression types, covering single-sentence and sentence-pair tasks. Contrary to prior assumptions that DA does not contribute to the enhancement of LMs' FT performance, our findings reveal that continued pre-training on augmented data can effectively improve the FT performance of the downstream tasks. In the most favourable case, continued pre-training improves the performance of FT by more than 10% in the few-shot learning setting. Our finding highlights the potential of DA as a powerful tool for bolstering LMs' performance.

翻译：近年来，语言模型在推动自然语言处理领域发展方面取得了显著进展。然而，数据增强技术对这些语言模型微调性能的影响一直存在争议。本研究评估了三种不同的微调方法与回译相结合在7项多样化的自然语言处理任务中的有效性，涵盖分类和回归类型，包括单句和句子对任务。与先前认为数据增强无助于提升语言模型微调性能的假设相反，我们的发现表明，在增强数据上进行持续预训练可以有效提升下游任务的微调性能。在最有利的情况下，持续预训练在小样本学习设置中将微调性能提升了超过10%。我们的发现强调了数据增强作为提升语言模型性能强大工具的潜力。