Large language models (LLMs) have exhibited great potential in autonomously completing tasks across real-world applications. Despite this, these LLM agents introduce unexpected safety risks when operating in interactive environments. Instead of centering on LLM-generated content safety in most prior studies, this work addresses the imperative need for benchmarking the behavioral safety of LLM agents within diverse environments. We introduce R-Judge, a benchmark crafted to evaluate the proficiency of LLMs in judging safety risks given agent interaction records. R-Judge comprises 162 agent interaction records, encompassing 27 key risk scenarios among 7 application categories and 10 risk types. It incorporates human consensus on safety with annotated safety risk labels and high-quality risk descriptions. Utilizing R-Judge, we conduct a comprehensive evaluation of 8 prominent LLMs commonly employed as the backbone for agents. The best-performing model, GPT-4, achieves 72.29% in contrast to the human score of 89.38%, showing considerable room for enhancing the risk awareness of LLMs. Notably, leveraging risk descriptions as environment feedback significantly improves model performance, revealing the importance of salient safety risk feedback. Furthermore, we design an effective chain of safety analysis technique to help the judgment of safety risks and conduct an in-depth case study to facilitate future research. R-Judge is publicly available at https://github.com/Lordog/R-Judge.

翻译：大语言模型（LLMs）在真实应用场景中展现出自主完成任务的巨大潜力。然而，这些LLM智能体在交互式环境中运行时可能引发不可预知的安全风险。不同于以往多数研究聚焦于LLM生成内容的安全性，本工作针对LLM智能体在多样环境中的行为安全基准评估这一迫切需求展开研究。我们提出R-Judge基准测试框架，用于评估LLMs通过智能体交互记录判断安全风险的能力。该基准包含162条智能体交互记录，涵盖7类应用中的27个关键风险场景及10种风险类型，同时整合了人类对安全性的共识标注（含安全风险标签与高质量风险描述）。基于R-Judge，我们对8种常用作智能体主干模型的代表性LLMs进行了全面评估。表现最优的GPT-4模型仅达72.29%的准确率，远低于人类基准89.38%，表明LLMs的风险感知能力仍有显著提升空间。值得注意的是，将风险描述作为环境反馈能显著提升模型性能，凸显了显著性安全风险反馈的重要性。此外，我们设计了有效的安全分析链技术辅助风险判断，并通过深度案例分析为后续研究提供参考。R-Judge现已开源：https://github.com/Lordog/R-Judge。