We present a method for systematically evaluating the correctness and robustness of instruction-tuned large language models (LLMs) for code generation via a new benchmark, Turbulence. Turbulence consists of a large set of natural language $\textit{question templates}$, each of which is a programming problem, parameterised so that it can be asked in many different forms. Each question template has an associated $\textit{test oracle}$ that judges whether a code solution returned by an LLM is correct. Thus, from a single question template, it is possible to ask an LLM a $\textit{neighbourhood}$ of very similar programming questions, and assess the correctness of the result returned for each question. This allows gaps in an LLM's code generation abilities to be identified, including $\textit{anomalies}$ where the LLM correctly solves $\textit{almost all}$ questions in a neighbourhood but fails for particular parameter instantiations. We present experiments against five LLMs from OpenAI, Cohere and Meta, each at two temperature configurations. Our findings show that, across the board, Turbulence is able to reveal gaps in LLM reasoning ability. This goes beyond merely highlighting that LLMs sometimes produce wrong code (which is no surprise): by systematically identifying cases where LLMs are able to solve some problems in a neighbourhood but do not manage to generalise to solve the whole neighbourhood, our method is effective at highlighting $\textit{robustness}$ issues. We present data and examples that shed light on the kinds of mistakes that LLMs make when they return incorrect code results.

翻译：我们提出了一种通过新基准测试“湍流”（Turbulence）系统化评估指令微调大语言模型（LLMs）在代码生成任务中正确性与鲁棒性的方法。湍流包含大量自然语言$\textit{问题模板}$，每个模板对应一个编程问题，并通过参数化使其能以多种不同形式被提出。每个问题模板关联一个$\textit{测试预言}$（test oracle），用于判断LLM返回的代码解决方案是否正确。因此，通过单一问题模板，可向LLM提出一个由高度相似的编程问题构成的$\textit{邻域}$（neighbourhood），并评估其对每个问题返回结果的正确性。这有助于识别LLM在代码生成能力中的不足，包括$\textit{异常现象}$——即LLM能正确解决邻域中$\textit{几乎全部}$问题，却对特定参数实例失败。我们针对来自OpenAI、Cohere和Meta的五种LLM（每种采用两个温度配置）进行了实验。结果表明，湍流能够全面揭示LLM推理能力的缺陷。这不仅仅是为了凸显LLM偶尔生成错误代码这一现象（这并不令人意外）：通过系统识别LLM能解决邻域部分问题但无法泛化至整个邻域的情况，我们的方法有效凸显了$\textit{鲁棒性}$问题。我们提供了数据和案例，阐明LLM在返回错误代码结果时常见错误类型。