Finding a best response policy is a central objective in game theory and multi-agent learning, with modern population-based training approaches employing reinforcement learning algorithms as best-response oracles to improve play against candidate opponents (typically previously learnt policies). We propose Best Response Expert Iteration (BRExIt), which accelerates learning in games by incorporating opponent models into the state-of-the-art learning algorithm Expert Iteration (ExIt). BRExIt aims to (1) improve feature shaping in the apprentice, with a policy head predicting opponent policies as an auxiliary task, and (2) bias opponent moves in planning towards the given or learnt opponent model, to generate apprentice targets that better approximate a best response. In an empirical ablation on BRExIt's algorithmic variants against a set of fixed test agents, we provide statistical evidence that BRExIt learns better performing policies than ExIt.

翻译：寻找最优应对策略是博弈论和多智能体学习中的核心目标，现代的基于种群训练方法采用强化学习算法作为最优应对预言机，以改进针对候选对手（通常是先前学习到的策略）的对局表现。我们提出了最优应对专家迭代（BRExIt），该方法通过将对手模型整合到最先进的专家迭代（ExIt）学习算法中，加速了游戏中的学习过程。BRExIt旨在：（1）在学徒中改进特征塑造，通过一个预测对手策略的策略头作为辅助任务；（2）在规划过程中将对手的移动偏向于给定或学习到的对手模型，从而生成更能近似最优应对的学徒目标。通过对BRExIt的算法变体针对一组固定测试智能体进行经验性消融分析，我们提供了统计证据，表明BRExIt学习到的策略性能优于ExIt。