Evaluating and improving the security capabilities of code agents requires high-quality, executable vulnerability tasks. However, existing works rely on costly, unscalable manual reproduction and suffer from outdated data distributions. To address these, we present CVE-Factory, the first multi-agent framework to achieve expert-level quality in automatically transforming sparse CVE metadata into fully executable agentic tasks. Cross-validation against human expert reproductions shows that CVE-Factory achieves 95\% solution correctness and 96\% environment fidelity, confirming its expert-level quality. It is also evaluated on the latest realistic vulnerabilities and achieves a 66.2\% verified success. This automation enables two downstream contributions. First, we construct LiveCVEBench, a continuously updated benchmark of 190 tasks spanning 14 languages and 153 repositories that captures emerging threats including AI-tooling vulnerabilities. Second, we synthesize over 1,000 executable training environments, the first large-scale scaling of agentic tasks in code security. Fine-tuned Qwen3-32B improves from 5.3\% to 35.8\% on LiveCVEBench, surpassing Claude 4.5 Sonnet, with gains generalizing to Terminal Bench (12.5\% to 31.3\%). We open-source CVE-Factory, LiveCVEBench, Abacus-cve (fine-tuned model), training dataset, and leaderboard. All resources are available at https://github.com/livecvebench/CVE-Factory .
翻译:评估和提升代码智能体的安全能力需要高质量、可执行的漏洞任务。然而,现有工作依赖于成本高昂且难以规模化的人工复现,并受限于过时的数据分布。为解决这些问题,我们提出了CVE-Factory,这是首个通过多智能体框架将稀疏的CVE元数据自动转化为完全可执行的智能体任务,并达到专家级质量的方法。与人类专家复现结果的交叉验证表明,CVE-Factory实现了95%的解决方案正确率和96%的环境保真度,证实了其专家级质量。在最新的现实漏洞评估中,其验证成功率也达到了66.2%。此项自动化能力带来了两项下游贡献。首先,我们构建了LiveCVEBench,这是一个持续更新的基准测试集,包含涵盖14种编程语言和153个代码库的190项任务,能够捕捉包括AI工具链漏洞在内的新兴威胁。其次,我们合成了超过1000个可执行的训练环境,首次实现了代码安全领域智能体任务的大规模扩展。经过微调的Qwen3-32B模型在LiveCVEBench上的表现从5.3%提升至35.8%,超越了Claude 4.5 Sonnet,且其提升效果可泛化至Terminal Bench(从12.5%提升至31.3%)。我们开源了CVE-Factory、LiveCVEBench、Abacus-cve(微调模型)、训练数据集及排行榜。所有资源均可在 https://github.com/livecvebench/CVE-Factory 获取。