The difficulty of appropriately assigning credit is particularly heightened in cooperative MARL with sparse reward, due to the concurrent time and structural scales involved. Automatic subgoal generation (ASG) has recently emerged as a viable MARL approach inspired by utilizing subgoals in intrinsically motivated reinforcement learning. However, end-to-end learning of complex task planning from sparse rewards without prior knowledge, undoubtedly requires massive training samples. Moreover, the diversity-promoting nature of existing ASG methods can lead to the "over-representation" of subgoals, generating numerous spurious subgoals of limited relevance to the actual task reward and thus decreasing the sample efficiency of the algorithm. To address this problem and inspired by the disentangled representation learning, we propose a novel "disentangled" decision-making method, Semantically Aligned task decomposition in MARL (SAMA), that prompts pretrained language models with chain-of-thought that can suggest potential goals, provide suitable goal decomposition and subgoal allocation as well as self-reflection-based replanning. Additionally, SAMA incorporates language-grounded RL to train each agent's subgoal-conditioned policy. SAMA demonstrates considerable advantages in sample efficiency compared to state-of-the-art ASG methods, as evidenced by its performance on two challenging sparse-reward tasks, Overcooked and MiniRTS.

翻译：在具有稀疏奖励的协作多智能体强化学习中，由于时间和结构尺度同时存在，信用分配的难度尤为突出。受内驱强化学习中使用子目标的启发，自动子目标生成近年来已成为一种可行的多智能体强化学习方法。然而，在无先验知识的情况下，从稀疏奖励中端到端学习复杂任务规划，无疑需要大量训练样本。此外，现有自动子目标生成方法中促进多样性的特性可能导致子目标的"过度表征"，生成大量与实际任务奖励相关性有限的无价值子目标，从而降低算法的样本效率。为解决这一问题并受解耦表征学习的启发，我们提出了一种新颖的"解耦"决策方法——多智能体强化学习中的语义对齐任务分解（SAMA），该方法利用链式思维提示预训练语言模型，使其能够建议潜在目标、提供合适的目标分解与子目标分配，以及基于自我反思的重新规划。此外，SAMA结合基于语言引导的强化学习来训练每个智能体的子目标条件策略。在两个具有挑战性的稀疏奖励任务（Overcooked与MiniRTS）上的实验表明，与最先进的自动子目标生成方法相比，SAMA在样本效率上展现出显著优势。