Few-shot learning with large-scale, pre-trained language models is a powerful way to answer questions about code, e.g., how to complete a given code example, or even generate code snippets from scratch. The success of these models raises the question whether they could serve as a basis for building a wide range code generation tools. Traditionally, such tools are built manually and separately for each task. Instead, few-shot learning may allow to obtain different tools from a single pre-trained language model by simply providing a few examples or a natural language description of the expected tool behavior. This paper studies to what extent a state-of-the-art, pre-trained language model of code, Codex, may serve this purpose. We consider three code manipulation and code generation tasks targeted by a range of traditional tools: (i) code mutation; (ii) test oracle generation from natural language documentation; and (iii) test case generation. For each task, we compare few-shot learning to a manually built tool. Our results show that the model-based tools complement (code mutation), are on par (test oracle generation), or even outperform their respective traditionally built tool (test case generation), while imposing far less effort to develop them. By comparing the effectiveness of different variants of the model-based tools, we provide insights on how to design an appropriate input ("prompt") to the model and what influence the size of the model has. For example, we find that providing a small natural language description of the code generation task is an easy way to improve predictions. Overall, we conclude that few-shot language models are surprisingly effective, yet there is still more work to be done, such as exploring more diverse ways of prompting and tackling even more involved tasks.

翻译：以大规模、 预先培训的语言模型进行少见的学习是回答有关代码问题的有力方法, 例如, 如何完成一个指定的代码示例, 甚至从零开始生成代码片断。 这些模型的成功提出了这样的问题: 这些工具能否成为构建范围广泛的代码生成工具的基础。 传统上, 这些工具是手工制造的, 并且为每个任务分别制作。 相反, 少见的学习可能允许从一个经过培训的单一语言模型中获得不同的工具, 仅仅提供几个例子或者自然语言描述预期工具行为。 本文研究了一个最先进的、 受过预先培训的语言模型, 或甚至从零开始生成代码。 我们考虑三个代码操作和代码生成任务能否成为构建范围很广的代码生成工具的基础。 (i) 代码突变;(ii) 从自然语言文档文档中测试或显示生成的生成过程; (iii) 测试案例生成过程。 对于每一项任务, 我们比较了少见的模型学习模式, 以手动工具仍然可以找到( 代码突变的) 最简单的( test orcle) 生成), 或者甚至比喻地分析一个传统设计工具的变异化方法。