Testing is an integral part of the software development process. Yet, writing tests is time-consuming and therefore often neglected. Classical test generation tools such as EvoSuite generate behavioral test suites by optimizing for coverage, but tend to produce tests that are hard to understand. Language models trained on code can generate code that is highly similar to that written by humans, but current models are trained to generate each file separately, as is standard practice in natural language processing, and thus fail to consider the code-under-test context when producing a test file. In this work, we propose the Aligned Code And Tests Language Model (CAT-LM), a GPT-style language model with 2.7 Billion parameters, trained on a corpus of Python and Java projects. We utilize a novel pretraining signal that explicitly considers the mapping between code and test files when available. We also drastically increase the maximum sequence length of inputs to 8,192 tokens, 4x more than typical code generation models, to ensure that the code context is available to the model when generating test code. We analyze its usefulness for realistic applications, showing that sampling with filtering (e.g., by compilability, coverage) allows it to efficiently produce tests that achieve coverage similar to ones written by developers while resembling their writing style. By utilizing the code context, CAT-LM generates more valid tests than even much larger language models trained with more data (CodeGen 16B and StarCoder) and substantially outperforms a recent test-specific model (TeCo) at test completion. Overall, our work highlights the importance of incorporating software-specific insights when training language models for code and paves the way to more powerful automated test generation.

翻译：测试是软件开发过程中不可或缺的一环。然而，编写测试用例耗时费力，因此常被忽视。传统的测试生成工具（如EvoSuite）通过优化覆盖率生成行为级测试套件，但往往产生难以理解的测试代码。基于代码训练的语言模型能生成与人类编写高度相似的代码，但目前模型均遵循自然语言处理领域的常规做法，即独立生成每个文件，因此在生成测试文件时未能考虑被测代码的上下文。本研究提出对齐代码与测试语言模型（CAT-LM），这是一个拥有27亿参数的GPT式语言模型，使用Python和Java项目的语料库进行训练。我们采用了一种创新的预训练信号，在代码与测试文件存在映射关系时明确考量该关联。同时，我们将输入的最大序列长度提升至8192个token，这是典型代码生成模型的4倍，以确保模型在生成测试代码时能获取代码上下文信息。通过分析其在实际应用中的效用，我们发现通过过滤采样（如基于可编译性、覆盖率等指标）能高效生成测试用例，其在实现与开发者编写的测试相近覆盖率的同时，保持了类似的写作风格。利用代码上下文，CAT-LM生成的测试用例甚至优于使用更多数据训练的更大规模语言模型（CodeGen 16B和StarCoder），并在测试补全任务上显著超越了近期专用于测试的模型（TeCo）。总体而言，本研究强调了在训练代码语言模型时融入软件工程领域知识的重要性，为更强大的自动化测试生成技术奠定了基础。