As large language models (LLMs) become integral to safety-critical applications, ensuring their robustness against adversarial prompts is paramount. However, existing red teaming datasets suffer from inconsistent risk categorizations, limited domain coverage, and outdated evaluations, hindering systematic vulnerability assessments. To address these challenges, we introduce RedBench, a universal dataset aggregating 37 benchmark datasets from leading conferences and repositories, comprising 29,362 samples across attack and refusal prompts. RedBench employs a standardized taxonomy with 22 risk categories and 19 domains, enabling consistent and comprehensive evaluations of LLM vulnerabilities. We provide a detailed analysis of existing datasets, establish baselines for modern LLMs, and open-source the dataset and evaluation code. Our contributions facilitate robust comparisons, foster future research, and promote the development of secure and reliable LLMs for real-world deployment. Code: https://github.com/knoveleng/redeval

翻译：随着大型语言模型（LLM）在安全关键型应用中的日益普及，确保其对抗对抗性提示的鲁棒性变得至关重要。然而，现有的红队测试数据集存在风险分类不一致、领域覆盖有限以及评估方法过时等问题，阻碍了系统性的漏洞评估。为解决这些挑战，我们提出了RedBench，这是一个通用数据集，它整合了来自顶级学术会议和公开仓库的37个基准数据集，包含攻击性提示和拒绝性提示共计29,362个样本。RedBench采用标准化的分类体系，涵盖22个风险类别和19个领域，能够对LLM漏洞进行一致且全面的评估。我们对现有数据集进行了详细分析，为现代LLM建立了性能基线，并开源了数据集和评估代码。我们的贡献有助于实现鲁棒的模型比较，推动未来研究，并促进开发适用于现实世界部署的安全可靠的大型语言模型。代码：https://github.com/knoveleng/redeval