In the context of multi-agent reinforcement learning, generalization is a challenge to solve various tasks that may require different joint policies or coordination without relying on policies specialized for each task. We refer to this type of problem as a multi-task, and we train agents to be versatile in this multi-task setting through a single training process. To address this challenge, we introduce TRajectory-class-Aware Multi-Agent reinforcement learning (TRAMA). In TRAMA, agents recognize a task type by identifying the class of trajectories they are experiencing through partial observations, and the agents use this trajectory awareness or prediction as additional information for action policy. To this end, we introduce three primary objectives in TRAMA: (a) constructing a quantized latent space to generate trajectory embeddings that reflect key similarities among them; (b) conducting trajectory clustering using these trajectory embeddings; and (c) building a trajectory-class-aware policy. Specifically for (c), we introduce a trajectory-class predictor that performs agent-wise predictions on the trajectory class; and we design a trajectory-class representation model for each trajectory class. Each agent takes actions based on this trajectory-class representation along with its partial observation for task-aware execution. The proposed method is evaluated on various tasks, including multi-task problems built upon StarCraft II. Empirical results show further performance improvements over state-of-the-art baselines.

翻译：在多智能体强化学习的背景下，泛化能力是解决各类任务的关键挑战，这些任务可能需要不同的联合策略或协作方式，且不依赖于针对每个任务专门设计的策略。我们将此类问题称为多任务问题，并通过单一训练过程使智能体在多任务场景中具备通用适应能力。为应对这一挑战，我们提出了轨迹类别感知的多智能体强化学习方法（TRAMA）。在TRAMA中，智能体通过局部观测识别所经历轨迹的类别来判定任务类型，并将这种轨迹感知或预测结果作为行动策略的附加信息。为此，我们在TRAMA中设定了三个核心目标：（a）构建量化潜在空间以生成能反映轨迹间关键相似性的轨迹嵌入；（b）利用这些轨迹嵌入进行轨迹聚类；（c）建立轨迹类别感知策略。特别针对目标（c），我们引入了轨迹类别预测器，该预测器能对轨迹类别进行智能体级别的预测；同时为每个轨迹类别设计了轨迹类别表征模型。每个智能体基于该轨迹类别表征及其局部观测信息采取行动，实现任务感知的决策执行。所提方法在多种任务上进行了评估，包括基于《星际争霸II》构建的多任务问题。实验结果表明，该方法相较于现有先进基线模型取得了进一步的性能提升。