While a large number of pre-trained models of source code have been successfully developed and applied to a variety of software engineering (SE) tasks in recent years, our understanding of these pre-trained models is arguably fairly limited. With the goal of advancing our understanding of these models, we perform the first systematic empirical comparison of 19 recently-developed pre-trained models of source code on 13 SE tasks. To gain additional insights into these models, we adopt a recently-developed 4-dimensional categorization of pre-trained models, and subsequently investigate whether there are correlations between different categories of pre-trained models and their performances on different SE tasks.

翻译：近年来，尽管大量源代码预训练模型已被成功开发并应用于多种软件工程任务，但我们对这些预训练模型的理解仍相当有限。为深化对这些模型的认识，我们首次对19个近期开发的源代码预训练模型在13个软件工程任务上进行了系统性实证比较。为获取更深入的见解，我们采用近期提出的四维预训练模型分类法，进而探究不同类别预训练模型与不同软件工程任务性能之间是否存在相关性。