We present a method for systematically evaluating the correctness and robustness of instruction-tuned large language models (LLMs) for code generation via a new benchmark, Turbulence. Turbulence consists of a large set of natural language $\textit{question templates}$, each of which is a programming problem, parameterised so that it can be asked in many different forms. Each question template has an associated $\textit{test oracle}$ that judges whether a code solution returned by an LLM is correct. Thus, from a single question template, it is possible to ask an LLM a $\textit{neighbourhood}$ of very similar programming questions, and assess the correctness of the result returned for each question. This allows gaps in an LLM's code generation abilities to be identified, including $\textit{anomalies}$ where the LLM correctly solves $\textit{almost all}$ questions in a neighbourhood but fails for particular parameter instantiations. We present experiments against five LLMs from OpenAI, Cohere and Meta, each at two temperature configurations. Our findings show that, across the board, Turbulence is able to reveal gaps in LLM reasoning ability. This goes beyond merely highlighting that LLMs sometimes produce wrong code (which is no surprise): by systematically identifying cases where LLMs are able to solve some problems in a neighbourhood but do not manage to generalise to solve the whole neighbourhood, our method is effective at highlighting $\textit{robustness}$ issues. We present data and examples that shed light on the kinds of mistakes that LLMs make when they return incorrect code results.

翻译：我们提出一种通过新基准Turbulence系统化评估指令调优大语言模型（LLMs）在代码生成任务中的正确性与鲁棒性的方法。Turbulence包含大量自然语言$\textit{问题模板}$，每个模板代表一个编程问题，通过参数化可生成多种不同提问形式。每个问题模板关联一个$\textit{测试预言}$，用于判断LLM返回的代码解是否正确。因此，从单个问题模板可向LLM提出一系列极其相似的编程问题（即$\textit{邻域}$），并评估其对每个问题的返回结果正确性。这使我们能够识别LLM代码生成能力的缺陷——包括$\textit{异常情形}$：当LLM能正确解答邻域中$\textit{几乎所有}$问题，却因特定参数实例化而失败时。我们针对OpenAI、Cohere和Meta的五个LLM在两种温度配置下进行实验。结果表明，Turbulence能够全面揭示LLM推理能力的缺陷。这不仅仅是简单指出LLM偶尔产生错误代码（这已不足为奇）：通过系统识别LLM能解决邻域内部分问题但无法泛化至整个邻域的案例，我们的方法有效凸显了$\textit{鲁棒性}$问题。我们提供的数据和实例揭示了LLM在返回错误代码结果时常见的错误类型。