Transformers have gained popularity in the software engineering (SE) literature. These deep learning models are usually pre-trained through a self-supervised objective, meant to provide the model with basic knowledge about a language of interest (e.g., Java). A classic pre-training objective is the masked language model (MLM), in which a percentage of tokens from the input (e.g., a Java method) is masked, with the model in charge of predicting them. Once pre-trained, the model is then fine-tuned to support the specific downstream task of interest (e.g., code summarization). While there is evidence suggesting the boost in performance provided by pre-training, little is known about the impact of the specific pre-training objective(s) used. Indeed, MLM is just one of the possible pre-training objectives and recent work from the natural language processing field suggest that pre-training objectives tailored for the specific downstream task of interest may substantially boost the model's performance. In this study, we focus on the impact of pre-training objectives on the performance of transformers when automating code-related tasks. We start with a systematic literature review aimed at identifying the pre-training objectives used in SE. Then, we pre-train 32 transformers using both (i) generic pre-training objectives usually adopted in SE; and (ii) pre-training objectives tailored to specific code-related tasks subject of our experimentation, namely bug-fixing, code summarization, and code completion. We also compare the pre-trained models with non pre-trained ones. Our results show that: (i) pre-training helps in boosting performance only if the amount of fine-tuning data available is small; (ii) the MLM objective is usually sufficient to maximize the prediction performance of the model, even when comparing it with pre-training objectives specialized for the downstream task at hand.

翻译：Transformer在软件工程（SE）文献中日益普及。这些深度学习模型通常通过自监督目标进行预训练，旨在为模型提供目标语言（如Java）的基础知识。经典的预训练目标是掩码语言模型（MLM），该方法将输入（如Java方法）中一定比例的标记进行掩码处理，由模型负责预测这些掩码内容。预训练完成后，模型会通过微调来支持特定的下游任务（如代码摘要）。尽管有证据表明预训练能提升性能，但特定预训练目标的影响仍鲜为人知。实际上，MLM只是可能的预训练目标之一，而自然语言处理领域的最新研究表明，针对特定下游任务定制的预训练目标可能显著提升模型性能。本研究聚焦于预训练目标对自动化代码相关任务中Transformer性能的影响。我们首先通过系统性文献综述梳理软件工程领域使用的预训练目标，随后基于以下两类目标预训练了32个Transformer模型：(i）软件工程中常用的通用预训练目标；(ii）针对实验所涉具体代码相关任务（即缺陷修复、代码摘要和代码补全）定制的预训练目标。我们还将预训练模型与未经预训练的模型进行了对比。结果表明：(i）仅当可用微调数据量较小时，预训练才能有效提升性能；(ii）即使与针对当前下游任务特化的预训练目标相比，MLM目标通常已足以最大化模型的预测性能。