Large language models (LLMs) achieve promising results in code generation based on a given natural language description. They have been integrated into open-source projects and commercial products to facilitate daily coding activities. The natural language description in the prompt is crucial for LLMs to comprehend users' requirements. Prior studies uncover that LLMs are sensitive to the changes in the prompts, including slight changes that look inconspicuous. However, the natural language descriptions often vary in real-world scenarios (e.g., different formats, grammar, and wording). Prior studies on the robustness of LLMs are often based on random perturbations and such perturbations may not actually happen. In this paper, we conduct a comprehensive study to investigate how are code LLMs robust to variations of natural language description in real-world scenarios. We summarize 18 categories of perturbations of natural language and 3 combinations of co-occurred categories based on our literature review and an online survey with practitioners. We propose an automated framework, NLPerturbator, which can perform perturbations of each category given a set of prompts. Through a series of experiments on code generation using six code LLMs, we find that the perturbed prompts can decrease the performance of code generation by a considerable margin (e.g., up to 21.2%, and 4.8% to 6.1% on average). Our study highlights the importance of enhancing the robustness of LLMs to real-world variations in the prompts, as well as the essentiality of attentively constructing the prompts.

翻译：大语言模型（LLMs）在基于给定自然语言描述生成代码方面取得了有前景的结果。它们已被集成到开源项目和商业产品中，以促进日常编码活动。提示中的自然语言描述对于LLMs理解用户需求至关重要。先前的研究揭示，LLMs对提示中的变化非常敏感，包括看似不显眼的细微改动。然而，在现实场景中，自然语言描述常常存在差异（例如，不同的格式、语法和措辞）。先前关于LLMs鲁棒性的研究通常基于随机扰动，而这些扰动可能实际上并不会发生。本文通过一项全面研究，探究代码LLMs在现实场景中对自然语言描述变异的鲁棒性。基于文献综述和对从业者的在线调查，我们总结了18类自然语言扰动以及3类共现类别的组合。我们提出了一个自动化框架NLPerturbator，该框架能够针对一组提示执行每个类别的扰动。通过对六种代码LLMs进行代码生成的一系列实验，我们发现扰动后的提示会显著降低代码生成性能（例如，最高下降21.2%，平均下降4.8%至6.1%）。我们的研究强调了增强LLMs对提示中现实世界变异的鲁棒性的重要性，以及精心构建提示的必要性。