Existing agent safety benchmarks report binary accuracy, conflating early intervention with post-mortem analysis. A detector that flags a violation at step 8 enables intervention; one that reports it at step 48 provides only forensic value. This distinction is critical, yet current benchmarks cannot measure it. We introduce StepShield, the first benchmark to evaluate when violations are detected, not just whether. StepShield contains 9,213 code agent trajectories, including 1,278 meticulously annotated training pairs and a 7,935-trajectory test set with a realistic 8.1% rogue rate. Rogue behaviors are grounded in real-world security incidents across six categories. We propose three novel temporal metrics: Early Intervention Rate (EIR), Intervention Gap, and Tokens Saved. Surprisingly, our evaluation reveals that an LLM-based judge achieves 59% EIR while a static analyzer achieves only 26%, a 2.3x performance gap that is entirely invisible to standard accuracy metrics. We further show that early detection has direct economic benefits: our cascaded HybridGuard detector reduces monitoring costs by 75% and projects to $108M in cumulative savings over five years at enterprise scale. By shifting the focus of evaluation from whether to when, StepShield provides a new foundation for building safer and more economically viable AI agents. The code and data are released under an Apache 2.0 license.

翻译：现有智能体安全基准报告二元准确率，将早期干预与事后分析混为一谈。在第8步标记违规的检测器可实现干预；而在第48步报告违规的检测器仅具备取证价值。这一区分至关重要，但现有基准无法对其进行衡量。我们提出StepShield——首个评估违规何时被检测而非仅是否被检测的基准。StepShield包含9,213条代码智能体轨迹，涵盖1,278条精细标注的训练样本对，以及一个包含7,935条轨迹、具有真实8.1%异常率的数据集。异常行为基于现实安全事件，涵盖六大类别。我们提出三项新颖的时序指标：早期干预率、干预间隔与节省标记数。令人惊讶的是，评估结果显示基于LLM的评判器达到59%的早期干预率，而静态分析器仅达到26%，这2.3倍的性能差距在标准准确率指标中完全无法显现。我们进一步证明早期检测具有直接经济效益：级联式HybridGuard检测器将监测成本降低75%，预计在企业级规模下五年累计可节省1.08亿美元。通过将评估重点从“是否”转向“何时”，StepShield为构建更安全且更具经济可行性的AI智能体提供了新基础。代码与数据均以Apache 2.0许可证发布。