Explaining multi-agent systems (MAS) is urgent as these systems become increasingly prevalent in various applications. Previous work has proveided explanations for the actions or states of agents, yet falls short in understanding the black-boxed agent's importance within a MAS and the overall team strategy. To bridge this gap, we propose EMAI, a novel agent-level explanation approach that evaluates the individual agent's importance. Inspired by counterfactual reasoning, a larger change in reward caused by the randomized action of agent indicates its higher importance. We model it as a MARL problem to capture interactions across agents. Utilizing counterfactual reasoning, EMAI learns the masking agents to identify important agents. Specifically, we define the optimization function to minimize the reward difference before and after action randomization and introduce sparsity constraints to encourage the exploration of more action randomization of agents during training. The experimental results in seven multi-agent tasks demonstratee that EMAI achieves higher fidelity in explanations than baselines and provides more effective guidance in practical applications concerning understanding policies, launching attacks, and patching policies.

翻译：随着多智能体系统（MAS）在各领域应用日益广泛，对其可解释性的需求变得尤为迫切。现有研究虽能解释智能体的行为或状态，却难以揭示黑盒智能体在MAS中的重要性及其对整体团队策略的贡献。为弥补这一不足，本文提出EMAI——一种新颖的智能体层级解释方法，用于评估个体智能体的重要性。该方法受反事实推理启发：若某智能体的随机化行为导致系统奖励产生较大变化，则表明该智能体具有更高重要性。我们将此建模为多智能体强化学习（MARL）问题以捕捉智能体间的交互关系。EMAI通过反事实推理学习对智能体进行掩码处理，从而识别关键智能体。具体而言，我们定义了优化函数以最小化行为随机化前后的奖励差异，并引入稀疏性约束以鼓励训练过程中探索更多智能体行为随机化方案。在七类多智能体任务中的实验结果表明，相较于基线方法，EMAI在解释保真度方面表现更优，并在策略理解、发起攻击和策略修复等实际应用中提供了更有效的指导。