Phishing has become a prominent risk in modern cybersecurity, often used to bypass technological defences by exploiting predictable human behaviour. Warning dialogues are a standard mitigation measure, but the lack of explanatory clarity and static content limits their effectiveness. In this paper, we report on our research to assess the capacity of Large Language Models (LLMs) to generate clear, concise, and scalable explanations for phishing warnings. We carried out a large-scale between-subjects user study (N = 750) to compare the influence of warning dialogues supplemented with manually generated explanations against those generated by two LLMs, Claude 3.5 Sonnet and Llama 3.3 70B. We investigated two explanatory styles (feature-based and counterfactual) for their effects on behavioural metrics (click-through rate) and perceptual outcomes (e.g., trust, risk, clarity). The results provide empirical evidence that LLM-generated explanations achieve a level of protection statistically comparable to expert-crafted messages, effectively automating a high-cost task. While Claude 3.5 Sonnet showed a trend towards reducing click-through rates compared to manual baselines, Llama 3.3, despite being perceived as clearer, did not yield the same behavioral benefits. Feature-based explanations were more effective for genuine phishing attempts, whereas counterfactual explanations diminished false-positive rates. Other variables, such as workload, gender, and prior familiarity with warning dialogues, significantly moderated the effectiveness of warnings. These results indicate that LLMs can be used to automatically build explanations for warning users against phishing, and that such solutions are scalable, adaptive, and consistent with human-centred values.

翻译：钓鱼攻击已成为现代网络安全中的突出风险，常通过利用可预测的人类行为来绕过技术防御。警告对话框是一种标准的缓解措施，但其解释清晰度不足和内容静态化限制了其有效性。本文报告了评估大型语言模型（LLMs）为钓鱼警告生成清晰、简洁且可扩展解释能力的研究。我们开展了一项大规模被试间用户研究（N = 750），比较了补充手动生成解释的警告对话框与由两种LLM（Claude 3.5 Sonnet 和 Llama 3.3 70B）生成解释的警告对话框的影响。我们研究了两种解释风格（基于特征的解释和反事实解释）对行为指标（点击率）和感知结果（如信任度、风险感知、清晰度）的影响。结果提供了实证证据，表明LLM生成的解释能达到与专家精心设计消息在统计上相当的保护水平，从而有效实现了这一高成本任务的自动化。虽然Claude 3.5 Sonnet在降低点击率方面相比人工基线显示出趋势，但Llama 3.3尽管被认为更清晰，却未产生相同的行为效益。基于特征的解释对真实钓鱼尝试更有效，而反事实解释则降低了误报率。其他变量，如工作负荷、性别及对警告对话框的先前熟悉程度，显著调节了警告的有效性。这些结果表明，LLMs可用于自动构建针对钓鱼攻击的用户警告解释，且此类解决方案具有可扩展性、适应性，并符合以人为中心的价值观。