In a multi-agent system, agents share their local observations to gain global situational awareness for decision making and collaboration using a message passing system. When to send a message, how to encode a message, and how to leverage the received messages directly affect the effectiveness of the collaboration among agents. When training a multi-agent cooperative game using reinforcement learning (RL), the message passing system needs to be optimized together with the agent policies. This consequently increases the model's complexity and poses significant challenges to the convergence and performance of learning. To address this issue, we propose the Belief-map Assisted Multi-agent System (BAMS), which leverages a neuro-symbolic belief map to enhance training. The belief map decodes the agent's hidden state to provide a symbolic representation of the agent's understanding of the environment and other agent's status. The simplicity of symbolic representation allows the gathering and comparison of the ground truth information with the belief, which provides an additional channel of feedback for the learning. Compared to the sporadic and delayed feedback coming from the reward in RL, the feedback from the belief map is more consistent and reliable. Agents using BAMS can learn a more effective message passing network to better understand each other, resulting in better performance in a cooperative predator and prey game with varying levels of map complexity and compare it to previous multi-agent message passing models. The simulation results showed that BAMS reduced training epochs by 66\%, and agents who apply the BAMS model completed the game with 34.62\% fewer steps on average.

翻译：在多智能体系统中，智能体通过消息传递系统共享局部观测信息，以获取全局态势感知，从而进行决策与协作。何时发送消息、如何编码消息以及如何利用接收到的消息，直接影响智能体间的协作效能。在使用强化学习训练多智能体协作博弈时，消息传递系统需与智能体策略共同优化，这增加了模型的复杂性，并对学习的收敛性与性能提出了重大挑战。为解决该问题，我们提出信念图辅助多智能体系统，该系统利用神经符号信念图来增强训练效果。信念图通过解码智能体的隐藏状态，提供其对环境及其他智能体状态的符号化表征。符号表征的简洁性使得能够收集真实信息并与信念进行比较，从而为学习提供额外的反馈通道。相较于强化学习中来自奖励的稀疏且延迟的反馈，信念图提供的反馈更为一致可靠。采用BAMS的智能体能够学习到更高效的消息传递网络，从而更好地相互理解，在具有不同地图复杂度的协作捕食者-猎物博弈中表现出更优性能，并与先前的多智能体消息传递模型进行了对比。仿真结果表明，BAMS将训练周期减少了66%，且应用BAMS模型的智能体完成游戏的平均步数减少了34.62%。