A code generation model generates code by taking a prompt from a code comment, existing code, or a combination of both. Although code generation models (e.g., GitHub Copilot) are increasingly being adopted in practice, it is unclear whether they can successfully be used for unit test generation without fine-tuning for a strongly typed language like Java. To fill this gap, we investigated how well three models (Codex, GPT-3.5-Turbo, and StarCoder) can generate unit tests. We used two benchmarks (HumanEval and Evosuite SF110) to investigate the effect of context generation on the unit test generation process. We evaluated the models based on compilation rates, test correctness, test coverage, and test smells. We found that the Codex model achieved above 80% coverage for the HumanEval dataset, but no model had more than 2% coverage for the EvoSuite SF110 benchmark. The generated tests also suffered from test smells, such as Duplicated Asserts and Empty Tests.

翻译：代码生成模型通过从代码注释、现有代码或两者的组合中获取提示来生成代码。尽管代码生成模型（例如GitHub Copilot）在实践中被越来越多地采用，但目前尚不清楚它们能否在不针对像Java这样的强类型语言进行微调的情况下成功用于单元测试生成。为了填补这一空白，我们研究了三种模型（Codex、GPT-3.5-Turbo和StarCoder）生成单元测试的能力。我们使用了两个基准测试（HumanEval和Evosuite SF110）来探究上下文生成对单元测试生成过程的影响。我们基于编译通过率、测试正确性、测试覆盖率和测试异味对模型进行了评估。结果发现，Codex模型在HumanEval数据集上达到了80%以上的覆盖率，但没有任何模型在EvoSuite SF110基准测试上获得超过2%的覆盖率。生成的测试还存在测试异味，例如重复断言和空测试。