Recently, fine-tuning pre-trained code models such as CodeBERT on downstream tasks has achieved great success in many software testing and analysis tasks. While effective and prevalent, fine-tuning the pre-trained parameters incurs a large computational cost. In this paper, we conduct an extensive experimental study to explore what happens to layer-wise pre-trained representations and their encoded code knowledge during fine-tuning. We then propose efficient alternatives to fine-tune the large pre-trained code model based on the above findings. Our experimental study shows that (1) lexical, syntactic and structural properties of source code are encoded in the lower, intermediate, and higher layers, respectively, while the semantic property spans across the entire model. (2) The process of fine-tuning preserves most of the code properties. Specifically, the basic code properties captured by lower and intermediate layers are still preserved during fine-tuning. Furthermore, we find that only the representations of the top two layers change most during fine-tuning for various downstream tasks. (3) Based on the above findings, we propose Telly to efficiently fine-tune pre-trained code models via layer freezing. The extensive experimental results on five various downstream tasks demonstrate that training parameters and the corresponding time cost are greatly reduced, while performances are similar or better. Replication package including source code, datasets, and online Appendix is available at: \url{https://github.com/DeepSoftwareAnalytics/Telly}.

翻译：近期，针对下游任务对预训练代码模型（如CodeBERT）进行微调已在许多软件测试与分析任务中取得巨大成功。尽管有效且广泛使用，但微调预训练参数带来了高昂的计算成本。本文中，我们开展了一项广泛的实验研究，探究微调过程中逐层预训练表示及其编码的代码知识如何变化。基于上述发现，我们提出了高效微调大型预训练代码模型的替代方案。实验研究表明：（1）源代码的词汇、句法和结构属性分别编码在下层、中层和上层，而语义属性则贯穿整个模型。（2）微调过程保留了大部分代码属性。具体而言，由下层和中层捕获的基本代码属性在微调期间仍保持不变。此外，我们发现对于各种下游任务，仅有顶部两层的表示在微调过程中变化最大。（3）基于上述发现，我们提出Telly，通过层冻结法高效微调预训练代码模型。在五个不同下游任务上的广泛实验结果表明，训练参数及相应时间成本大幅降低，同时性能保持相似或更优。包含源代码、数据集及在线附录的复现包可在以下地址获取：\url{https://github.com/DeepSoftwareAnalytics/Telly}。