Large Language Models have demonstrated a remarkable capability in natural language and program generation and software development. However, the source code generated by the LLMs does not always meet quality requirements and may fail to compile. Therefore, many studies evolve into agents that can reason about the problem before generating the source code for the solution. The goal of this paper is to study the degree to which such agents benefit from access to software development tools, in our case, a \texttt{gcc} compiler. We conduct a computational experiment on the RosettaCode dataset, on 699 programming tasks in C. We evaluate how the integration with a compiler shifts the role of the language model from a passive generator to an active agent capable of iteratively developing runnable programs based on feedback from the compiler. We evaluated 16 language models with sizes ranging from small (135 million) to medium (3 billion) and large (70 billion). Our results show that access to a compiler improved the compilation success by 5.3 to 79.4 percentage units in compilation without affecting the semantics of the generated program. Syntax errors dropped by 75\%, and errors related to undefined references dropped by 87\% for the tasks where the agents outperformed the baselines. We also observed that in some cases, smaller models with a compiler outperform larger models with a compiler. We conclude that it is essential for LLMs to have access to software engineering tools to enhance their performance and reduce the need for large models in software engineering, such as reducing our energy footprint.

翻译：大语言模型在自然语言处理、程序生成和软件开发方面展现出卓越能力。然而，LLM生成的源代码并不总能满足质量要求，且可能无法通过编译。因此，许多研究演化为能够先对问题进行推理、再生成解决方案源代码的智能体。本文旨在探究此类智能体从软件开发工具（本研究中为\texttt{gcc}编译器）的访问权限中获益的程度。我们在RosettaCode数据集上对699项C语言编程任务进行了计算实验，评估编译器集成如何将语言模型从被动生成器转变为能够基于编译器反馈迭代开发可运行程序的主动智能体。我们评估了16个参数量从小型（1.35亿）到中型（30亿）及大型（700亿）不等的语言模型。实验结果表明，编译器访问使编译成功率提升5.3至79.4个百分点，且不影响生成程序的语义。在智能体表现优于基线的任务中，语法错误减少75%，未定义引用错误减少87%。我们还观察到在某些情况下，配备编译器的小型模型性能可超越配备编译器的大型模型。本研究得出结论：为LLM提供软件工程工具对提升其性能至关重要，并能降低软件工程中对大型模型的依赖，例如减少能源消耗。