Many automated test generation techniques have been developed to aid developers with writing tests. To facilitate full automation, most existing techniques aim to either increase coverage, or generate exploratory inputs. However, existing test generation techniques largely fall short of achieving more semantic objectives, such as generating tests to reproduce a given bug report. Reproducing bugs is nonetheless important, as our empirical study shows that the number of tests added in open source repositories due to issues was about 28% of the corresponding project test suite size. Meanwhile, due to the difficulties of transforming the expected program semantics in bug reports into test oracles, existing failure reproduction techniques tend to deal exclusively with program crashes, a small subset of all bug reports. To automate test generation from general bug reports, we propose LIBRO, a framework that uses Large Language Models (LLMs), which have been shown to be capable of performing code-related tasks. Since LLMs themselves cannot execute the target buggy code, we focus on post-processing steps that help us discern when LLMs are effective, and rank the produced tests according to their validity. Our evaluation of LIBRO shows that, on the widely studied Defects4J benchmark, LIBRO can generate failure reproducing test cases for 33% of all studied cases (251 out of 750), while suggesting a bug reproducing test in first place for 149 bugs. To mitigate data contamination, we also evaluate LIBRO against 31 bug reports submitted after the collection of the LLM training data terminated: LIBRO produces bug reproducing tests for 32% of the studied bug reports. Overall, our results show LIBRO has the potential to significantly enhance developer efficiency by automatically generating tests from bug reports.

翻译：许多自动化测试生成技术已被开发出来，以帮助开发人员编写测试。为实现完全自动化，现有大多数技术旨在提高覆盖率或生成探索性输入。然而，现有测试生成技术在很大程度上未能实现更具语义性的目标，例如生成测试用例以复现给定的缺陷报告。缺陷复现仍然重要，因为我们的实证研究表明，由于问题而添加到开源仓库中的测试数量约占对应项目测试套件规模的28%。同时，由于难以将缺陷报告中的预期程序语义转化为测试断言，现有失效复现技术往往仅处理程序崩溃这一小类缺陷报告。为自动化从通用缺陷报告生成测试，我们提出LIBRO框架，该框架使用已被证明能执行代码相关任务的大语言模型。由于大语言模型本身无法执行目标缺陷代码，我们聚焦于后处理步骤，以帮助辨别大语言模型何时有效，并根据生成测试的有效性对其进行排序。我们在广泛研究的Defects4J基准上的评估表明，LIBRO能为33%的研究案例（750个中的251个）生成失效复现测试用例，同时为149个缺陷优先推荐缺陷复现测试。为缓解数据污染，我们还在大语言模型训练数据收集截止后提交的31份缺陷报告上评估LIBRO：LIBRO为32%的研究缺陷报告生成了缺陷复现测试。总体而言，我们的结果表明，LIBRO具有通过从缺陷报告自动生成测试来显著提升开发人员效率的潜力。