Stochastic partial observability poses a major challenge for decentralized coordination in multi-agent reinforcement learning but is largely neglected in state-of-the-art research due to a strong focus on state-based centralized training for decentralized execution (CTDE) and benchmarks that lack sufficient stochasticity like StarCraft Multi-Agent Challenge (SMAC). In this paper, we propose Attention-based Embeddings of Recurrence In multi-Agent Learning (AERIAL) to approximate value functions under stochastic partial observability. AERIAL replaces the true state with a learned representation of multi-agent recurrence, considering more accurate information about decentralized agent decisions than state-based CTDE. We then introduce MessySMAC, a modified version of SMAC with stochastic observations and higher variance in initial states, to provide a more general and configurable benchmark regarding stochastic partial observability. We evaluate AERIAL in Dec-Tiger as well as in a variety of SMAC and MessySMAC maps, and compare the results with state-based CTDE. Furthermore, we evaluate the robustness of AERIAL and state-based CTDE against various stochasticity configurations in MessySMAC.

翻译：随机部分可观测性是多智能体强化学习中分散协调面临的主要挑战，但由于当前研究主要集中在基于状态的集中训练与分散执行（CTDE）范式，以及缺乏足够随机性的基准测试（如星际争霸多智能体挑战赛SMAC），该问题在很大程度上被忽视。本文提出多智能体学习中的注意力机制循环嵌入（AERIAL）方法，用于在随机部分可观测性下逼近价值函数。AERIAL用多智能体循环的学习表示替代真实状态，相比基于状态的CTDE，能更准确地考虑分散智能体决策信息。我们进一步引入MessySMAC——SMAC的改进版本，通过增加状态初始方差和随机观测，构建更通用且可配置的随机部分可观测性基准测试。在Dec-Tiger以及多种SMAC和MessySMAC地图上评估AERIAL，并与基于状态的CTDE进行对比；此外，还评估了AERIAL和基于状态的CTDE在MessySMAC不同随机性配置下的鲁棒性。