Safety evaluation of large language models (LLMs) increasingly relies on LLM-as-a-Judge frameworks, but the high cost of frontier models limits scalability. We propose a cost-efficient multi-agent judging framework that employs Small Language Models (SLMs) through structured debates among critic, defender, and judge agents. To rigorously assess safety judgments, we construct HAJailBench, a large-scale human-annotated jailbreak benchmark comprising 12,000 adversarial interactions across diverse attack methods and target models. The dataset provides fine-grained, expert-labeled ground truth for evaluating both safety robustness and judge reliability. Our SLM-based framework achieves agreement comparable to GPT-4o judges on HAJailBench while substantially reducing inference cost. Ablation results show that three rounds of debate yield the optimal balance between accuracy and efficiency. These findings demonstrate that structured, value-aligned debate enables SLMs to capture semantic nuances of jailbreak attacks and that HAJailBench offers a reliable foundation for scalable LLM safety evaluation.

翻译：大语言模型（LLM）的安全性评估日益依赖LLM即法官框架，但前沿模型的高昂成本限制了其可扩展性。我们提出了一种成本高效的多智能体评判框架，通过批评者、辩护者和法官智能体之间的结构化辩论，利用小语言模型（SLM）进行安全评估。为严格评估安全判断，我们构建了HAJailBench——一个大规模人工标注的越狱基准数据集，包含涵盖多种攻击方法和目标模型的12,000次对抗性交互。该数据集提供了细粒度、专家标注的真实标签，用于评估安全鲁棒性和法官可靠性。我们基于SLM的框架在HAJailBench上实现了与GPT-4o法官相当的判断一致性，同时显著降低了推理成本。消融实验表明，三轮辩论能在准确性与效率之间达到最优平衡。这些发现证明：结构化、价值对齐的辩论能使SLM捕捉越狱攻击的语义细微差别，而HAJailBench为可扩展的LLM安全评估提供了可靠基础。