Opponent modelling has proven effective in enhancing the decision-making of the controlled agent by constructing models of opponent agents. However, existing methods often rely on access to the observations and actions of opponents, a requirement that is infeasible when such information is either unobservable or challenging to obtain. To address this issue, we introduce Distributional Opponent-aided Multi-agent Actor-Critic (DOMAC), the first speculative opponent modelling algorithm that relies solely on local information (i.e., the controlled agent's observations, actions, and rewards). Specifically, the actor maintains a speculated belief about the opponents using the tailored speculative opponent models that predict the opponents' actions using only local information. Moreover, DOMAC features distributional critic models that estimate the return distribution of the actor's policy, yielding a more fine-grained assessment of the actor's quality. This thus more effectively guides the training of the speculative opponent models that the actor depends upon. Furthermore, we formally derive a policy gradient theorem with the proposed opponent models. Extensive experiments under eight different challenging multi-agent benchmark tasks within the MPE, Pommerman and StarCraft Multiagent Challenge (SMAC) demonstrate that our DOMAC successfully models opponents' behaviours and delivers superior performance against state-of-the-art methods with a faster convergence speed.

翻译：对手建模通过构建对手代理的模型，已被证明能有效增强受控代理的决策能力。然而，现有方法通常需要访问对手的观测和动作，当此类信息不可观测或难以获取时，这一要求难以实现。为解决此问题，我们提出了分布式对手辅助多智能体演员-评论家（DOMAC），这是首个仅依赖局部信息（即受控代理的观测、动作和奖励）的推测性对手建模算法。具体而言，演员利用定制化的推测对手模型（仅通过局部信息预测对手动作）维持对对手的推测信念。此外，DOMAC采用分布式评论家模型，通过估计演员策略的回报分布，形成对演员质量的更精细评估，从而更有效地指导演员所依赖的推测对手模型的训练。进一步地，我们基于所提出的对手模型正式推导了策略梯度定理。在MPE、Pommerman及星际争霸多智能体挑战（SMAC）的八项不同难度多智能体基准任务上的大量实验表明，我们的DOMAC成功建模了对手行为，并以更快的收敛速度实现了优于现有最先进方法的性能。