Recent advances in Competitive Self-Play (CSP) have achieved, or even surpassed, human level performance in complex game environments such as Dota 2 and StarCraft II using Distributed Multi-Agent Reinforcement Learning (MARL). One core component of these methods relies on creating a pool of learning agents -- consisting of the Main Agent, past versions of this agent, and Exploiter Agents -- where Exploiter Agents learn counter-strategies to the Main Agents. A key drawback of these approaches is the large computational cost and physical time that is required to train the system, making them impractical to deploy in highly iterative real-life settings such as video game productions. In this paper, we propose the Minimax Exploiter, a game theoretic approach to exploiting Main Agents that leverages knowledge of its opponents, leading to significant increases in data efficiency. We validate our approach in a diversity of settings, including simple turn based games, the arcade learning environment, and For Honor, a modern video game. The Minimax Exploiter consistently outperforms strong baselines, demonstrating improved stability and data efficiency, leading to a robust CSP-MARL method that is both flexible and easy to deploy.

翻译：近期，竞争性自我对弈（CSP）领域的进展，通过分布式多智能体强化学习（MARL），在《Dota 2》和《星际争霸II》等复杂游戏环境中已实现甚至超越人类水平的表现。这些方法的核心之一在于构建一个学习智能体池——包含主智能体、该智能体的历史版本以及利用者智能体——其中利用者智能体学习针对主智能体的反制策略。然而，这些方法的关键缺陷在于训练系统所需的高昂计算成本和物理时间，使其难以部署于高度迭代的现实场景（如电子游戏制作）。本文提出极小化极大利用者（Minimax Exploiter）这一基于博弈论的方法，通过利用对手信息来对抗主智能体，显著提升数据效率。我们在多样化的环境中验证了该方法，包括简单回合制游戏、街机学习环境以及现代电子游戏《荣耀战魂》。极小化极大利用者持续优于强基线方法，展现出更优的稳定性与数据效率，从而形成一种既灵活又易于部署的鲁棒性CSP-MARL方法。