Language models are promising solutions for tackling increasing complex problems. In software engineering, they recently attracted attention in code assistants, with programs automatically written in a given programming language from a programming task description in natural language. They have the potential to save time and effort when writing code. However, these systems are currently poorly understood, preventing them from being used optimally. In this paper, we investigate the various input parameters of two language models, and conduct a study to understand if variations of these input parameters (e.g. programming task description and the surrounding context, creativity of the language model, number of generated solutions) can have a significant impact on the quality of the generated programs. We design specific operators for varying input parameters and apply them over two code assistants (Copilot and Codex) and two benchmarks representing algorithmic problems (HumanEval and LeetCode). Our results showed that varying the input parameters can significantly improve the performance of language models. However, there is a tight dependency when varying the temperature, the prompt and the number of generated solutions, making potentially hard for developers to properly control the parameters to obtain an optimal result. This work opens opportunities to propose (automated) strategies for improving performance.

翻译：语言模型是解决日益复杂问题的有前途方案。在软件工程领域，它们最近作为代码助手引人注目，能够根据自然语言描述的程序设计任务，自动生成指定编程语言的代码。这有望节省编写代码的时间和精力。然而，这些系统目前尚未被充分理解，阻碍了其最优利用。本文研究了两种语言模型的各种输入参数，并通过实验探究这些输入参数的变化（例如，编程任务描述及其周围上下文、语言模型的创造性、生成方案数量）是否会对生成代码的质量产生显著影响。我们设计了用于改变输入参数的专用算子，并将其应用于两个代码助手（Copilot与Codex）以及两个代表算法问题的基准测试（HumanEval与LeetCode）。结果表明，调整输入参数可显著提升语言模型的性能。然而，温度、提示词与生成方案数量的调整之间存在紧密依赖性，这可能使开发人员难以恰当控制参数以获得最优结果。本研究为提出（自动化）性能改进策略开辟了机遇。