Unit tests play a key role in ensuring the correctness of software. However, manually creating unit tests is a laborious task, motivating the need for automation. This paper presents TestPilot, an adaptive test generation technique that leverages Large Language Models (LLMs). TestPilot uses Codex, an off-the-shelf LLM, to automatically generate unit tests for a given program without requiring additional training or few-shot learning on examples of existing tests. In our approach, Codex is provided with prompts that include the signature and implementation of a function under test, along with usage examples extracted from documentation. If a generated test fails, TestPilot's adaptive component attempts to generate a new test that fixes the problem by re-prompting the model with the failing test and error message. We created an implementation of TestPilot for JavaScript and evaluated it on 25 npm packages with a total of 1,684 API functions to generate tests for. Our results show that the generated tests achieve up to 93.1% statement coverage (median 68.2%). Moreover, on average, 58.5% of the generated tests contain at least one assertion that exercises functionality from the package under test. Our experiments with excluding parts of the information included in the prompts show that all components contribute towards the generation of effective test suites. Finally, we find that TestPilot does not generate memorized tests: 92.7% of our generated tests have $\leq$ 50% similarity with existing tests (as measured by normalized edit distance), with none of them being exact copies.

翻译：单元测试在确保软件正确性方面发挥关键作用。然而，手工创建单元测试是一项耗时的工作，促使了自动化需求的产生。本文提出TestPilot，一种利用大型语言模型（LLMs）的自适应测试生成技术。TestPilot使用Codex（一种现成的大型语言模型）为给定程序自动生成单元测试，无需额外训练或对现有测试示例进行少样本学习。在我们的方法中，为Codex提供的提示包含被测函数的签名和实现，以及从文档中提取的使用示例。如果生成的测试失败，TestPilot的自适应组件会通过向模型重新提供失败测试和错误信息的提示，尝试生成修复问题的新测试。我们为JavaScript实现了TestPilot，并在25个npm包（共含1,684个API函数）上进行了评估。结果表明，生成的测试语句覆盖率最高可达93.1%（中位数68.2%）。此外，平均有58.5%的生成测试包含至少一个断言，用于验证被测包的功能。通过排除提示中包含的部分信息进行实验，我们发现所有组件都有助于生成有效的测试套件。最后，TestPilot并不会生成记忆性测试：92.7%的生成测试与现有测试的相似度≤50%（按归一化编辑距离计算），且无任何测试与现有测试完全一致。