A code generation model generates code by taking a prompt from a code comment, existing code, or a combination of both. Although code generation models (e.g., GitHub Copilot) are increasingly being adopted in practice, it is unclear whether they can successfully be used for unit test generation without fine-tuning for a strongly typed language like Java. To fill this gap, we investigated how well three models (Codex, GPT-3.5-Turbo, and StarCoder) can generate unit tests. We used two benchmarks (HumanEval and Evosuite SF110) to investigate the effect of context generation on the unit test generation process. We evaluated the models based on compilation rates, test correctness, test coverage, and test smells. We found that the Codex model achieved above 80% coverage for the HumanEval dataset, but no model had more than 2% coverage for the EvoSuite SF110 benchmark. The generated tests also suffered from test smells, such as Duplicated Asserts and Empty Tests.

翻译：代码生成模型通过从代码注释、现有代码或两者结合中提取提示来生成代码。尽管代码生成模型（如GitHub Copilot）在实践中日益普及，但对于强类型语言（如Java），在不进行微调的情况下，它们能否成功用于单元测试生成仍不明确。为填补这一空白，我们研究了三种模型（Codex、GPT-3.5-Turbo和StarCoder）生成单元测试的能力。采用两个基准测试（HumanEval和Evosuite SF110），探究上下文生成对单元测试生成过程的影响。我们基于编译通过率、测试正确性、测试覆盖率和测试异味对模型进行了评估。结果表明，Codex模型在HumanEval数据集上实现了80%以上的覆盖率，但所有模型在EvoSuite SF110基准上的覆盖率均不足2%。生成的测试还存在重复断言和空测试等测试异味问题。