Security vulnerability repair is a difficult task that is in dire need of automation. Two groups of techniques have shown promise: (1) large code language models (LLMs) that have been pre-trained on source code for tasks such as code completion, and (2) automated program repair (APR) techniques that use deep learning (DL) models to automatically fix software bugs. This paper is the first to study and compare Java vulnerability repair capabilities of LLMs and DL-based APR models. The contributions include that we (1) apply and evaluate five LLMs (Codex, CodeGen, CodeT5, PLBART and InCoder), four fine-tuned LLMs, and four DL-based APR techniques on two real-world Java vulnerability benchmarks (Vul4J and VJBench), (2) design code transformations to address the training and test data overlapping threat to Codex, (3) create a new Java vulnerability repair benchmark VJBench, and its transformed version VJBench-trans and (4) evaluate LLMs and APR techniques on the transformed vulnerabilities in VJBench-trans. Our findings include that (1) existing LLMs and APR models fix very few Java vulnerabilities. Codex fixes 10.2 (20.4%), the most number of vulnerabilities. (2) Fine-tuning with general APR data improves LLMs' vulnerability-fixing capabilities. (3) Our new VJBench reveals that LLMs and APR models fail to fix many Common Weakness Enumeration (CWE) types, such as CWE-325 Missing cryptographic step and CWE-444 HTTP request smuggling. (4) Codex still fixes 8.3 transformed vulnerabilities, outperforming all the other LLMs and APR models on transformed vulnerabilities. The results call for innovations to enhance automated Java vulnerability repair such as creating larger vulnerability repair training data, tuning LLMs with such data, and applying code simplification transformation to facilitate vulnerability repair.

翻译：安全漏洞修复是一项迫切需要自动化的艰巨任务。目前两类技术展现出前景：（1）在源代码上预训练用于代码补全等任务的大规模代码语言模型（LLM），以及（2）利用深度学习（DL）模型自动修复软件缺陷的自动程序修复（APR）技术。本文首次研究并比较了大型代码语言模型与基于深度学习的APR模型在Java漏洞修复方面的能力。主要贡献包括：（1）在两个真实Java漏洞基准测试集（Vul4J和VJBench）上应用并评估了五个大型代码语言模型（Codex、CodeGen、CodeT5、PLBART和InCoder）、四个微调后的LLM以及四种基于深度学习的APR技术；（2）设计了代码变换方法以应对Codex训练数据与测试数据重叠的威胁；（3）创建了新的Java漏洞修复基准测试集VJBench及其变换版本VJBench-trans；（4）评估了LLM和APR技术在VJBench-trans变换漏洞上的表现。研究发现：（1）现有LLM和APR模型修复的Java漏洞极少，其中Codex修复了10.2个（20.4%），数量最多；（2）使用通用APR数据进行微调可提升LLM的漏洞修复能力；（3）新创建的VJBench基准测试表明，LLM和APR模型无法修复许多常见弱点枚举（CWE）类型，如CWE-325缺少加密步骤和CWE-444 HTTP请求走私；（4）Codex仍能修复8.3个变换后的漏洞，在变换漏洞上优于所有其他LLM和APR模型。这些结果表明需要创新方法以增强自动化Java漏洞修复能力，例如构建更大规模的漏洞修复训练数据、使用此类数据微调LLM，以及应用代码简化变换以促进漏洞修复。