We ran 300 repeated vulnerability-finding scans to measure how repeatable agentic large language model (LLM) security review is on the same JavaScript code, prompt, and benchmark harness. The headline result is that LLM security findings were unevenly repeatable: reference-matched findings were stable, but extra model reports varied heavily from run to run. Across 250 model runs, 80 of 161 unique unmatched findings appeared in only one of five identical repetitions, while only 22 appeared in all five. By contrast, when Claude matched a Snyk Code reference finding, the behavior was much more stable: 134 of 158 unique reference-matched findings appeared in all five repetitions. The benchmark also shows complementarity. Models consistently found familiar, high-signal exploit shapes, and in one case surfaced a likely Snyk Code product gap. Snyk Code static application security testing (SAST) was deterministic and better at systematically enumerating repeated data-flow sinks. The results support combining agentic LLM review with deterministic SAST rather than treating either technique as a replacement for the other.

翻译：我们运行了300次重复的漏洞发现扫描，以衡量在相同的JavaScript代码、提示词和基准测试框架下，智能体大语言模型安全审查的可重复性。主要结果表明，LLM的安全发现结果并非均匀可重复：与参考结果匹配的发现是稳定的，但模型额外报告的内容在不同运行之间差异很大。在250次模型运行中，161个唯一的未匹配发现中有80个仅在五次相同重复中的一次出现，而只有22个在所有五次中出现。相比之下，当Claude匹配到Snyk Code的参考发现时，行为稳定得多：158个唯一的参考匹配发现中有134个在所有五次重复中出现。该基准测试还显示了互补性。模型一致地找到了熟悉的、高信号利用模式，并在一个案例中揭示了Snyk Code可能存在的产品空白。Snyk Code静态应用安全测试具有确定性，且更擅长系统地枚举重复的数据流汇聚点。研究结果支持将智能体LLM审查与确定性SAST相结合，而非将任何一种技术视为另一种的替代品。