Opponent modeling methods typically involve two crucial steps: building a belief distribution over opponents' strategies, and exploiting this opponent model by playing a best response. However, existing approaches typically require domain-specific heurstics to come up with such a model, and algorithms for approximating best responses are hard to scale in large, imperfect information domains. In this work, we introduce a scalable and generic multiagent training regime for opponent modeling using deep game-theoretic reinforcement learning. We first propose Generative Best Respoonse (GenBR), a best response algorithm based on Monte-Carlo Tree Search (MCTS) with a learned deep generative model that samples world states during planning. This new method scales to large imperfect information domains and can be plug and play in a variety of multiagent algorithms. We use this new method under the framework of Policy Space Response Oracles (PSRO), to automate the generation of an \emph{offline opponent model} via iterative game-theoretic reasoning and population-based training. We propose using solution concepts based on bargaining theory to build up an opponent mixture, which we find identifying profiles that are near the Pareto frontier. Then GenBR keeps updating an \emph{online opponent model} and reacts against it during gameplay. We conduct behavioral studies where human participants negotiate with our agents in Deal-or-No-Deal, a class of bilateral bargaining games. Search with generative modeling finds stronger policies during both training time and test time, enables online Bayesian co-player prediction, and can produce agents that achieve comparable social welfare and Nash bargaining score negotiating with humans as humans trading among themselves.

翻译：对手建模方法通常涉及两个关键步骤：构建关于对手策略的信念分布，以及通过执行最优反应来利用该对手模型。然而，现有方法通常需要领域特定的启发式规则来构建此类模型，且近似最优反应的算法难以扩展到大规模不完全信息领域。本研究提出一种可扩展且通用的多智能体训练框架，利用深度博弈论强化学习进行对手建模。我们首先提出基于生成式最优反应（GenBR）的算法，该算法采用蒙特卡洛树搜索（MCTS）与学习到的深度生成模型，在规划过程中对世界状态进行采样。这一新方法可扩展至大规模不完全信息领域，并能以即插即用方式集成于多种多智能体算法中。我们将该新方法应用于策略空间响应范式（PSRO）框架下，通过迭代博弈论推理与基于种群的训练，自动生成离线对手模型。我们提出基于谈判理论的解概念来构建对手混合策略，发现该策略能识别接近帕累托前沿的剖面。随后，GenBR持续更新在线对手模型，并在游戏过程中对其进行动态应对。我们开展行为实验，让人类参与者在双边谈判游戏"Deal-or-No-Deal"中与智能体进行谈判。基于生成模型进行搜索能在训练与测试阶段发现更强的策略，支持在线贝叶斯协同预测，并能生成在与人类谈判时达到与人类内部交易相当的社会福利与纳什谈判得分的智能体。