Achieving and maintaining cooperation between agents to accomplish a common objective is one of the central goals of Multi-Agent Reinforcement Learning (MARL). Nevertheless in many real-world scenarios, separately trained and specialized agents are deployed into a shared environment, or the environment requires multiple objectives to be achieved by different coexisting parties. These variations among specialties and objectives are likely to cause mixed motives that eventually result in a social dilemma where all the parties are at a loss. In order to resolve this issue, we propose the Incentive Q-Flow (IQ-Flow) algorithm, which modifies the system's reward setup with an incentive regulator agent such that the cooperative policy also corresponds to the self-interested policy for the agents. Unlike the existing methods that learn to incentivize self-interested agents, IQ-Flow does not make any assumptions about agents' policies or learning algorithms, which enables the generalization of the developed framework to a wider array of applications. IQ-Flow performs an offline evaluation of the optimality of the learned policies using the data provided by other agents to determine cooperative and self-interested policies. Next, IQ-Flow uses meta-gradient learning to estimate how policy evaluation changes according to given incentives and modifies the incentive such that the greedy policy for cooperative objective and self-interested objective yield the same actions. We present the operational characteristics of IQ-Flow in Iterated Matrix Games. We demonstrate that IQ-Flow outperforms the state-of-the-art incentive design algorithm in Escape Room and 2-Player Cleanup environments. We further demonstrate that the pretrained IQ-Flow mechanism significantly outperforms the performance of the shared reward setup in the 2-Player Cleanup environment.

翻译：实现并维持智能体之间的合作以完成共同目标，是多智能体强化学习（Multi-Agent Reinforcement Learning, MARL）的核心目标之一。然而，在许多现实场景中，分别训练且具备专业功能的智能体被部署到共享环境中，或环境要求由不同共存的参与方实现多个目标。这些专业分工与目标之间的差异可能引发混合动机，最终导致所有参与方均蒙受损失的社会困境。为解决这一问题，我们提出激励Q流（Incentive Q-Flow, IQ-Flow）算法，该算法通过引入激励调控智能体修改系统的奖励设置，使得合作策略与智能体的自利策略保持一致。与现有学习激励自利智能体的方法不同，IQ-Flow不假设智能体的策略或学习算法，从而使所开发框架能够泛化至更广泛的应用场景。IQ-Flow利用其他智能体提供的数据，对所学策略的最优性进行离线评估，以确定合作策略与自利策略。随后，IQ-Flow采用元梯度学习估计策略评估如何随给定激励变化，并修改激励使得针对合作目标与自利目标的贪婪策略产生相同动作。我们在迭代矩阵博弈中展示了IQ-Flow的运行特性，并证明其在密室逃脱与双人清理环境中优于当前最先进的激励设计算法。进一步实验表明，预训练的IQ-Flow机制在双人清理环境中的表现显著优于共享奖励设置。