Many popular programming languages, including C#, Java, and Python, support exceptions. Exceptions are thrown during program execution if an unwanted event happens, e.g., a method is invoked with an illegal argument value. Software developers write exceptional behavior tests (EBTs) to check that their code detects unwanted events and throws appropriate exceptions. Prior research studies have shown the importance of EBTs, but those studies also highlighted that developers put most of their efforts on "happy paths", e.g., paths without unwanted events. To help developers fill the gap, we present the first framework, dubbed exLong, that automatically generates EBTs. exLong is a large language model instruction-tuned from CodeLlama and embeds reasoning about traces that lead to throw statements, conditional expressions that guard throw statements, and non-exceptional behavior tests that execute similar traces. We compare exLong with the state-of-the-art models for test generation (CAT-LM) and one of the strongest foundation models (GPT3.5), as well as with analysis-based tools for test generation (Randoop and EvoSuite). Our results show that exLong outperforms existing models and tools. Furthermore, we contributed several pull requests to open-source projects and 23 EBTs generated by exLong were already accepted.

翻译：包括C#、Java和Python在内的多种主流编程语言均支持异常处理机制。当程序执行过程中发生非预期事件（例如方法调用时传入非法参数值）时，系统将抛出异常。软件开发人员通过编写异常行为测试来验证代码能否准确检测非预期事件并抛出相应异常。已有研究证实了异常行为测试的重要性，但同时也指出开发者往往将主要精力集中于"正常执行路径"（即不包含非预期事件的路径）。为协助开发者弥补这一不足，我们提出了首个自动化生成异常行为测试的框架exLong。该框架基于CodeLlama进行指令微调的大语言模型，嵌入了对以下三要素的推理机制：导致throw语句的执行轨迹、守卫throw语句的条件表达式，以及执行相似轨迹的非异常行为测试。我们将exLong与当前最先进的测试生成模型（CAT-LM）、性能领先的基础模型（GPT3.5）以及基于分析的测试生成工具（Randoop和EvoSuite）进行了对比实验。结果表明exLong在性能上优于现有模型与工具。此外，我们向多个开源项目提交了若干拉取请求，其中已有23个由exLong生成的异常行为测试被正式采纳。