The difficulty of appropriately assigning credit is particularly heightened in cooperative MARL with sparse reward, due to the concurrent time and structural scales involved. Automatic subgoal generation (ASG) has recently emerged as a viable MARL approach inspired by utilizing subgoals in intrinsically motivated reinforcement learning. However, end-to-end learning of complex task planning from sparse rewards without prior knowledge, undoubtedly requires massive training samples. Moreover, the diversity-promoting nature of existing ASG methods can lead to the "over-representation" of subgoals, generating numerous spurious subgoals of limited relevance to the actual task reward and thus decreasing the sample efficiency of the algorithm. To address this problem and inspired by the disentangled representation learning, we propose a novel "disentangled" decision-making method, Semantically Aligned task decomposition in MARL (SAMA), that prompts pretrained language models with chain-of-thought that can suggest potential goals, provide suitable goal decomposition and subgoal allocation as well as self-reflection-based replanning. Additionally, SAMA incorporates language-grounded RL to train each agent's subgoal-conditioned policy. SAMA demonstrates considerable advantages in sample efficiency compared to state-of-the-art ASG methods, as evidenced by its performance on two challenging sparse-reward tasks, Overcooked and MiniRTS.

翻译：在稀疏奖励的协作多智能体强化学习中，由于时间和结构尺度的并发性，信用分配问题尤为突出。基于内在动机强化学习中的子目标启发，自动子目标生成（ASG）近期成为可行的多智能体强化学习方法。然而，在无先验知识的情况下从稀疏奖励中端到端学习复杂任务规划，无疑需要大量训练样本。此外，现有ASG方法促进多样性的特性可能导致子目标"过表征"，产生众多与实际任务奖励关联有限的虚假子目标，从而降低算法样本效率。针对该问题，受解耦表示学习启发，我们提出一种新颖的"解耦"决策方法——语义对齐的多智能体强化学习任务分解（SAMA），该方法利用链式思维提示预训练语言模型，使其能够建议潜在目标、提供适当的目标分解和子目标分配，以及基于自我反思的重新规划。同时，SAMA融合了语言引导的强化学习来训练每个智能体的子目标条件策略。与最先进的ASG方法相比，SAMA在两个具有挑战性的稀疏奖励任务（Overcooked和MiniRTS）上的表现证明了其在样本效率方面的显著优势。