Software engineering research has always being concerned with the improvement of code completion approaches, which suggest the next tokens a developer will likely type while coding. The release of GitHub Copilot constitutes a big step forward, also because of its unprecedented ability to automatically generate even entire functions from their natural language description. While the usefulness of Copilot is evident, it is still unclear to what extent it is robust. Specifically, we do not know the extent to which semantic-preserving changes in the natural language description provided to the model have an effect on the generated code function. In this paper we present an empirical study in which we aim at understanding whether different but semantically equivalent natural language descriptions result in the same recommended function. A negative answer would pose questions on the robustness of deep learning (DL)-based code generators since it would imply that developers using different wordings to describe the same code would obtain different recommendations. We asked Copilot to automatically generate 892 Java methods starting from their original Javadoc description. Then, we generated different semantically equivalent descriptions for each method both manually and automatically, and we analyzed the extent to which predictions generated by Copilot changed. Our results show that modifying the description results in different code recommendations in ~46% of cases. Also, differences in the semantically equivalent descriptions might impact the correctness of the generated code ~28%.

翻译：软件工程研究始终关注代码补全方法的改进，这些方法能预测开发者在编码时可能键入的下一个标记。GitHub Copilot的发布标志着重大进步，尤其是因为它具备前所未有的能力：能够根据自然语言描述自动生成完整函数。尽管Copilot的实用性显而易见，但其鲁棒性程度仍不明确。具体来说，我们尚不清楚提供给模型的自然语言描述中那些保持语义一致的变化，会在多大程度上影响生成的代码函数。本文提出一项实证研究，旨在探究不同但语义等价（semantically equivalent）的自然语言描述是否会导致相同的推荐函数。若答案是否定的，则将引发对基于深度学习（DL）的代码生成器鲁棒性的质疑，因为这意味着开发者在描述相同代码时使用不同的措辞会得到不同的推荐结果。我们要求Copilot基于原始Javadoc描述自动生成892个Java方法。随后，我们通过手动和自动方式为每个方法生成多种语义等价的描述，并分析Copilot生成的预测结果发生变化的程度。结果表明，约46%的情况下，修改描述会导致不同的代码推荐。此外，语义等价描述的差异可能在约28%的情况下影响生成代码的正确性。