Learning anticipation in Multi-Agent Reinforcement Learning (MARL) is a reasoning paradigm where agents anticipate the learning steps of other agents to improve cooperation among themselves. As MARL uses gradient-based optimization, learning anticipation requires using Higher-Order Gradients (HOG), with so-called HOG methods. Existing HOG methods are based on policy parameter anticipation, i.e., agents anticipate the changes in policy parameters of other agents. Currently, however, these existing HOG methods have only been applied to differentiable games or games with small state spaces. In this work, we demonstrate that in the case of non-differentiable games with large state spaces, existing HOG methods do not perform well and are inefficient due to their inherent limitations related to policy parameter anticipation and multiple sampling stages. To overcome these problems, we propose Off-Policy Action Anticipation (OffPA2), a novel framework that approaches learning anticipation through action anticipation, i.e., agents anticipate the changes in actions of other agents, via off-policy sampling. We theoretically analyze our proposed OffPA2 and employ it to develop multiple HOG methods that are applicable to non-differentiable games with large state spaces. We conduct a large set of experiments and illustrate that our proposed HOG methods outperform the existing ones regarding efficiency and performance.

翻译：多智能体强化学习中的学习预测是一种推理范式，其中智能体预测其他智能体的学习步骤以改善彼此间的协作。由于多智能体强化学习采用基于梯度的优化方法，学习预测需要使用高阶梯度方法。现有的高阶梯度方法基于策略参数预测，即智能体预测其他智能体策略参数的变化。然而，目前这些高阶梯度方法仅适用于可微分游戏或状态空间较小的游戏。在本工作中，我们证明在具有大状态空间的不可微分游戏中，现有的高阶梯度方法由于其在策略参数预测和多阶段采样方面的固有局限性，表现不佳且效率低下。为解决这些问题，我们提出了离线策略动作预测（OffPA2），这是一种新颖的框架，通过动作预测（即智能体预测其他智能体动作的变化）结合离线策略采样来实现学习预测。我们从理论上分析了所提出的OffPA2，并利用其开发了多种适用于具有大状态空间的不可微分游戏的高阶梯度方法。通过大量实验，我们证明所提出的高阶梯度方法在效率和性能上均优于现有方法。