System prompt configuration can make the difference between near-total phishing blindness and near-perfect detection in LLM email agents. We present PhishNChips, a study of 11 models under 10 prompt strategies, showing that prompt-model interaction is a first-order security variable: a single model's phishing bypass rate ranges from under 1% to 97% depending on how it is configured, while the false-positive cost of the same prompt varies sharply across models. We then show that optimizing prompts around highly predictive signals can improve benchmark performance, reaching up to 93.7% recall at 3.8% false positive rate, but also creates a brittle attack surface. In particular, domain-matching strategies perform well when legitimate emails mostly have matched sender and URL domains, yet degrade sharply when attackers invert that signal by registering matching infrastructure. Response-trace analysis shows that 98% of successful bypasses reason in ways consistent with the inverted signal: the models are following the instruction, but the instruction's core assumption has become false. A counter-intuitive corollary follows: making prompts more specific can degrade already-capable models by replacing broader multi-signal reasoning with exploitable single-signal dependence. We characterize the resulting tension between detection, usability, and adversarial robustness as a navigable tradeoff, introduce Safetility, a deployability-aware metric that penalizes false positives, and argue that closing the adversarial gap likely requires tool augmentation with external ground truth.

翻译：系统提示词配置可导致LLM邮件代理从近乎完全无法识别钓鱼攻击转变为近乎完美检测。我们提出PhishNChips研究，涵盖10种提示策略下的11个模型，表明提示词与模型的交互是一阶安全变量：同一模型根据配置不同，其钓鱼绕过率可从1%以下变化至97%，而相同提示词的误报成本在不同模型间差异显著。我们进一步证明，围绕高预测性信号优化提示词可提升基准性能（召回率达93.7%，误报率仅3.8%），但同时也造成了脆弱的攻击面。具体而言，当合法邮件大多具有匹配的发件人和URL域时，域匹配策略表现良好，但当攻击者通过注册匹配基础设施逆转该信号时，性能急剧下降。响应轨迹分析显示，98%的成功绕过遵循与逆转信号一致的推理模式：模型确实遵循了指令，但指令的核心假设已失效。一个反直觉推论随之产生：使提示词更具体反而可能削弱已有较强能力的模型，因为这将替代广泛的多信号推理，转而依赖可被利用的单信号。我们将检测、可用性与对抗鲁棒性之间的张力量化为可权衡的折中，引入Safetility——一种可感知部署性的指标（惩罚误报），并主张缩小对抗差距可能需要结合外部真实世界信息的工具增强。