Security vulnerability repair is a difficult task that is in dire need of automation. Two groups of techniques have shown promise: (1) large code language models (LLMs) that have been pre-trained on source code for tasks such as code completion, and (2) automated program repair (APR) techniques that use deep learning (DL) models to automatically fix software bugs. This paper is the first to study and compare Java vulnerability repair capabilities of LLMs and DL-based APR models. The contributions include that we (1) apply and evaluate five LLMs (Codex, CodeGen, CodeT5, PLBART and InCoder), four fine-tuned LLMs, and four DL-based APR techniques on two real-world Java vulnerability benchmarks (Vul4J and VJBench), (2) design code transformations to address the training and test data overlapping threat to Codex, (3) create a new Java vulnerability repair benchmark VJBench, and its transformed version VJBench-trans and (4) evaluate LLMs and APR techniques on the transformed vulnerabilities in VJBench-trans. Our findings include that (1) existing LLMs and APR models fix very few Java vulnerabilities. Codex fixes 10.2 (20.4%), the most number of vulnerabilities. (2) Fine-tuning with general APR data improves LLMs' vulnerability-fixing capabilities. (3) Our new VJBench reveals that LLMs and APR models fail to fix many Common Weakness Enumeration (CWE) types, such as CWE-325 Missing cryptographic step and CWE-444 HTTP request smuggling. (4) Codex still fixes 8.3 transformed vulnerabilities, outperforming all the other LLMs and APR models on transformed vulnerabilities. The results call for innovations to enhance automated Java vulnerability repair such as creating larger vulnerability repair training data, tuning LLMs with such data, and applying code simplification transformation to facilitate vulnerability repair.

翻译：安全漏洞修复是一项亟需自动化的艰巨任务。两类技术展现出前景：(1) 在源代码上预训练用于代码补全等任务的大型代码语言模型（LLMs），以及(2) 使用深度学习（DL）模型自动修复软件缺陷的自动化程序修复（APR）技术。本文首次研究并比较了LLMs与基于DL的APR模型在Java漏洞修复方面的能力。贡献包括：(1) 在两个真实世界的Java漏洞基准（Vul4J和VJBench）上，应用并评估了五个LLMs（Codex、CodeGen、CodeT5、PLBART和InCoder）、四个微调后的LLMs以及四个基于DL的APR技术；(2) 设计代码变换以应对Codex的训练数据与测试数据重叠的威胁；(3) 创建了一个新的Java漏洞修复基准VJBench及其变换版本VJBench-trans；(4) 评估LLMs和APR技术在VJBench-trans中变换后漏洞上的性能。我们的发现包括：(1) 现有的LLMs和APR模型能修复的Java漏洞极少。Codex修复了10.2个（20.4%）漏洞，数量最多。(2) 使用通用APR数据进行微调可提升LLMs的漏洞修复能力。(3) 我们的新VJBench揭示，LLMs和APR模型未能修复许多常见弱点枚举（CWE）类型，例如CWE-325（缺失加密步骤）和CWE-444（HTTP请求走私）。(4) Codex仍然修复了8.3个变换后的漏洞，在所有其他LLMs和APR模型中表现最佳。这些结果呼吁创新以增强自动化Java漏洞修复，例如创建更大的漏洞修复训练数据、使用此类数据微调LLMs，以及应用代码简化变换以促进漏洞修复。