Code generation with Large Language Models (LLMs) has been extensively studied and achieved remarkable progress. As a complementary aspect to code generation, test case generation is of crucial importance in ensuring the quality and reliability of code. However, using LLMs as test case generators has been much less explored. Current research along this line primarily focuses on enhancing code generation with assistance from test cases generated by LLMs, while the performance of LLMs in test case generation alone has not been comprehensively examined. To bridge this gap, we conduct extensive experiments to study how well LLMs can generate high-quality test cases. We find that as the problem difficulty increases, state-of-the-art LLMs struggle to generate correct test cases, largely due to their inherent limitations in computation and reasoning. To mitigate this issue, we further propose a multi-agent framework called \emph{TestChain} that decouples the generation of test inputs and test outputs. Notably, TestChain uses a ReAct format conversation chain for LLMs to interact with a Python interpreter in order to provide more accurate test outputs. Our results indicate that TestChain outperforms the baseline by a large margin. Particularly, in terms of the accuracy of test cases, TestChain using GPT-4 as the backbone achieves a 13.84\% improvement over the baseline on the LeetCode-hard dataset.

翻译：基于大型语言模型（LLMs）的代码生成已被广泛研究并取得了显著进展。作为代码生成的补充方面，测试用例生成在确保代码质量和可靠性方面至关重要。然而，将LLMs用作测试用例生成器的研究尚不充分。当前相关研究主要聚焦于通过LLMs生成的测试用例来增强代码生成，而LLMs单独生成测试用例的性能尚未得到全面评估。为弥补这一不足，我们开展了大量实验，系统探究LLMs生成高质量测试用例的能力。研究发现，随着问题难度的增加，最先进的LLMs在生成正确测试用例时面临困难，这主要源于其计算和推理能力的固有局限。为解决该问题，我们进一步提出了一种名为TestChain的多智能体框架，该框架将测试输入和测试输出生成过程解耦。值得注意的是，TestChain采用ReAct格式的对话链，使LLMs能够与Python解释器交互，从而生成更精确的测试输出。实验结果表明，TestChain在性能上大幅优于基线方法。特别是在测试用例准确性方面，基于GPT-4的TestChain在LeetCode-hard数据集上相比基线方法取得了13.84%的提升。