Deployable Large Language Models (LLMs) must conform to the criterion of helpfulness and harmlessness, thereby achieving consistency between LLMs outputs and human values. Red-teaming techniques constitute a critical way towards this criterion. Existing work rely solely on manual red team designs and heuristic adversarial prompts for vulnerability detection and optimization. These approaches lack rigorous mathematical formulation, thus limiting the exploration of diverse attack strategy within quantifiable measure and optimization of LLMs under convergence guarantees. In this paper, we present Red-teaming Game (RTG), a general game-theoretic framework without manual annotation. RTG is designed for analyzing the multi-turn attack and defense interactions between Red-team language Models (RLMs) and Blue-team Language Model (BLM). Within the RTG, we propose Gamified Red-teaming Solver (GRTS) with diversity measure of the semantic space. GRTS is an automated red teaming technique to solve RTG towards Nash equilibrium through meta-game analysis, which corresponds to the theoretically guaranteed optimization direction of both RLMs and BLM. Empirical results in multi-turn attacks with RLMs show that GRTS autonomously discovered diverse attack strategies and effectively improved security of LLMs, outperforming existing heuristic red-team designs. Overall, RTG has established a foundational framework for red teaming tasks and constructed a new scalable oversight technique for alignment.

翻译：可部署的大型语言模型（LLMs）必须符合有用性与无害性标准，从而确保LLMs输出与人类价值观的一致性。红队测试技术是实现该标准的关键途径。现有工作仅依赖人工红队设计与启发式对抗提示进行漏洞检测与优化，这些方法缺乏严谨的数学形式化，因此难以在可量化度量下探索多样化攻击策略，也无法在收敛性保证下优化LLMs。本文提出红队游戏（RTG）——一种无需人工标注的通用博弈论框架。RTG用于分析红队语言模型（RLMs）与蓝队语言模型（BLM）之间的多轮攻防交互。在RTG框架内，我们提出了引入语义空间多样性度量的博弈化红队求解器（GRTS）。GRTS是一种自动化红队技术，通过元博弈分析求解RTG的纳什均衡，对应RLMs与BLM理论上有保证的优化方向。使用RLMs进行多轮攻击的实验结果表明，GRTS能自主发现多样化攻击策略，有效提升LLMs的安全性，性能优于现有启发式红队设计。总体而言，RTG为红队测试任务建立了基础性框架，并构建了一种可扩展的对齐监督新技术。