Pre-trained language models are effective in a variety of natural language tasks, but it has been argued their capabilities fall short of fully learning meaning or understanding language. To understand the extent to which language models can learn some form of meaning, we investigate their ability to capture semantics of code beyond superficial frequency and co-occurrence. In contrast to previous research on probing models for linguistic features, we study pre-trained models in a setting that allows for objective and straightforward evaluation of a model's ability to learn semantics. In this paper, we examine whether such models capture the semantics of code, which is precisely and formally defined. Through experiments involving the manipulation of code fragments, we show that code pre-trained models of code learn a robust representation of the computational semantics of code that goes beyond superficial features of form alone

翻译：预训练语言模型在多种自然语言任务中表现有效，但已有观点认为其能力尚未完全达到学习意义或理解语言的程度。为探究语言模型在何种程度上能够学习某种形式的意义，我们研究了它们捕捉代码语义的能力，这种能力超越了表面的频率和共现统计。与先前对语言特征进行探测模型的研究不同，我们在一个允许客观直接评估模型语义学习能力的设置下研究预训练模型。本文通过涉及代码片段操纵的实验，检验了此类模型是否能够捕捉代码的语义——后者具有精确且形式化的定义。实验结果表明，代码预训练模型学习了代码计算语义的鲁棒表征，该表征超越了代码形式层面的表面特征。