Large language models (LLMs) can autonomously conduct multi-stage cyber attacks, but the consistency of their offensive behavior under repeated trials remains unstudied. This work presents the first large-scale empirical measurement of LLM attack consistency: 400 autonomous penetration testing runs (4 models, 100 each) against an identical honeypot hosting OWASP Juice Shop and two additional vulnerable services, holding prompt, orchestrator, and target constant. No model emitted a content refusal that survived the orchestrator's one-shot authorization re-prompt at iterations 0-1. Claude Sonnet 4's API calls did encounter upstream service unavailability - 91 of 1,135 calls returned HTTP 529 overloaded_error during a documented Anthropic capacity event, truncating 39 of 100 Claude runs. An earlier draft catalogued these as safety refusals; on full-log audit they are upstream API failures, not model-level refusals. Despite this, Claude achieved full exploitation in 61 of 100 runs; Gemini 2.5 Flash-Lite in 85; GPT-4o-mini in 56 while deploying 98 unique attack strategies; qwen2.5-coder:14b in 25. Failure modes are model-distinctive: Claude through API truncation (39 runs), qwen through premature completion (52), GPT-4o-mini through iteration-budget exhaustion (23). Cross-service credential reuse appeared only in configurations retaining the most conversation history (qwen 57%, GPT-4o-mini 49%, cloud models 0% on 5-exchange windows). Cross-model exploitation rate differences are statistically significant (p < 0.001) with large effect sizes; qwen vs. Gemini SQL injection rates differ at Cohen's h = 1.12. First-exploit timing fell within a 15-30 second wall-clock range. To our knowledge, this is the first study to measure autonomous LLM attack behavior at N=100 per model across a multi-service target.

翻译：大型语言模型（LLMs）能够自主执行多阶段网络攻击，但其在重复试验中攻击行为的一致性尚未被探究。本研究首次对LLM攻击一致性进行大规模实证测量：针对托管OWASP Juice Shop及两个额外脆弱服务的同一蜜罐（保持提示词、编排器和目标不变），执行400次自主渗透测试（4个模型各100次）。在迭代0-1阶段，所有模型均未输出经过编排器一次性授权重新提示后仍保留的内容拒斥。Claude Sonnet 4的API调用遭遇上游服务不可用情况——在记录的Anthropic容量事件期间，1135次调用中有91次返回HTTP 529 overloaded_error，导致100次Claude运行中的39次被截断。早期草稿将其归类为安全拒斥；经完整日志审计确认，这是上游API故障而非模型级拒斥。尽管如此，Claude在61/100次运行中实现完全利用；Gemini 2.5 Flash-Lite为85次；GPT-4o-mini在部署98种独特攻击策略下完成56次；qwen2.5-coder:14b完成25次。失败模式具有模型特异性：Claude因API截断（39次运行），qwen因过早完成（52次），GPT-4o-mini因迭代预算耗尽（23次）。跨服务凭证重用仅出现在保留最多对话历史的配置中（5轮窗口内，qwen 57%、GPT-4o-mini 49%、云端模型0%）。跨模型利用率的差异具有统计学显著性（p<0.001）且效应量较大；qwen与Gemini的SQL注入率差异Cohen's h=1.12。首次利用时间集中在15-30秒真实时间窗口内。据我们所知，这是首个在每种模型N=100且覆盖多服务目标条件下测量自主LLM攻击行为的研究。