Large language models (LLMs) have shown considerable promise for automated unit test generation, yet their practical effectiveness relative to human-written tests remains poorly understood. Existing evaluations commonly rely on coverage-oriented benchmarks that do not assess fault-detection capability directly. We present an empirical comparison of LLM-generated and human-written unit tests across three complementary Python benchmarks: 29 real historical bugs from BugsInPy, a function-level benchmark drawn from python-slugify and packaging, and a controlled paired benchmark. Our generation pipeline couples Gemini 2.5 Flash with a lightweight lexical retrieval mechanism that supplies bug-relevant context at generation time. Across eight quality dimensions, LLM-generated tests with retrieval-augmented context detect faults in 69% of cases compared to 17.2% for general-purpose human-written tests (Fisher's exact, $p < 0.001$, Cohen's $h = 1.10$). Critically, line and branch coverage are nearly identical between the two approaches (84.8% vs. 88.5% and 75.2% vs. 82.1%), confirming that coverage is an insufficient proxy for fault-detection capability. We discuss the conditions under which each approach excels, characterize their complementary strengths, and identify the critical role of retrieval context and reproducible benchmark construction in meaningful test-quality evaluation.

翻译：大型语言模型（LLM）在自动化单元测试生成方面展现出巨大潜力，但其与人类编写测试相比的实际有效性仍缺乏深入理解。现有评估通常依赖仅衡量覆盖率的基准测试，未直接评估缺陷检测能力。我们通过三个互补的Python基准测试对LLM生成测试与人类编写测试进行实证比较：BugsInPy中的29个真实历史缺陷、来自python-slugify和packaging的函数级基准测试，以及受控配对基准测试。我们的生成流程将Gemini 2.5 Flash与轻量级词汇检索机制相结合，在生成时提供缺陷相关上下文。在八个质量维度上，采用检索增强上下文的LLM生成测试在69%的案例中检测到缺陷，而通用人类编写测试仅为17.2%（Fisher精确检验，$p < 0.001$，Cohen's $h = 1.10$）。关键的是，两种方法的行覆盖率和分支覆盖率几乎相同（84.8% vs. 88.5%和75.2% vs. 82.1%），证实覆盖率不足以作为缺陷检测能力的代理指标。我们讨论了每种方法表现最佳的条件，描述了它们的互补优势，并指出了检索上下文与可复现基准构建在有意义的测试质量评估中的关键作用。