A code generation model generates code by taking a prompt from a code comment, existing code, or a combination of both. Although code generation models (e.g., GitHub Copilot) are increasingly being adopted in practice, it is unclear whether they can successfully be used for unit test generation without fine-tuning. To fill this gap, we investigated how well three generative models (CodeGen, Codex, and GPT-3.5) can generate test cases. We used two benchmarks (HumanEval and Evosuite SF110) to investigate the context generation's effect in the unit test generation process. We evaluated the models based on compilation rates, test correctness, coverage, and test smells. We found that the Codex model achieved above 80% coverage for the HumanEval dataset, but no model had more than 2% coverage for the EvoSuite SF110 benchmark. The generated tests also suffered from test smells, such as Duplicated Asserts and Empty Tests.

翻译：代码生成模型通过接收来自代码注释、现有代码或两者结合的提示来生成代码。尽管代码生成模型（例如 GitHub Copilot）在实践中被越来越多地采用，但不明确的是，这些模型是否能在无需微调的情况下成功用于单元测试生成。为填补这一空白，我们调研了三种生成模型（CodeGen、Codex 和 GPT-3.5）生成测试用例的能力。我们使用了两个基准测试（HumanEval 和 Evosuite SF110）来探究上下文生成在单元测试生成过程中的影响。我们基于编译成功率、测试正确性、覆盖率和测试坏味道对这些模型进行了评估。结果发现，Codex 模型在 HumanEval 数据集上达到了 80% 以上的覆盖率，但没有任何模型在 EvoSuite SF110 基准测试上的覆盖率超过 2%。生成的测试还出现了测试坏味道，例如重复断言和空测试。