Achieving and maintaining cooperation between agents to accomplish a common objective is one of the central goals of Multi-Agent Reinforcement Learning (MARL). Nevertheless in many real-world scenarios, separately trained and specialized agents are deployed into a shared environment, or the environment requires multiple objectives to be achieved by different coexisting parties. These variations among specialties and objectives are likely to cause mixed motives that eventually result in a social dilemma where all the parties are at a loss. In order to resolve this issue, we propose the Incentive Q-Flow (IQ-Flow) algorithm, which modifies the system's reward setup with an incentive regulator agent such that the cooperative policy also corresponds to the self-interested policy for the agents. Unlike the existing methods that learn to incentivize self-interested agents, IQ-Flow does not make any assumptions about agents' policies or learning algorithms, which enables the generalization of the developed framework to a wider array of applications. IQ-Flow performs an offline evaluation of the optimality of the learned policies using the data provided by other agents to determine cooperative and self-interested policies. Next, IQ-Flow uses meta-gradient learning to estimate how policy evaluation changes according to given incentives and modifies the incentive such that the greedy policy for cooperative objective and self-interested objective yield the same actions. We present the operational characteristics of IQ-Flow in Iterated Matrix Games. We demonstrate that IQ-Flow outperforms the state-of-the-art incentive design algorithm in Escape Room and 2-Player Cleanup environments. We further demonstrate that the pretrained IQ-Flow mechanism significantly outperforms the performance of the shared reward setup in the 2-Player Cleanup environment.

翻译：实现并维持智能体间的合作以完成共同目标是多智能体强化学习（MARL）的核心目标之一。然而在许多现实场景中，独立训练的专业化智能体被部署到共享环境中，或环境需要不同共存方实现多个目标。这些专业化分工与目标差异可能引发混合动机，最终导致各方均受损失的"社会困境"。为解决该问题，我们提出激励Q流（IQ-Flow）算法，通过激励调节智能体来修改系统的奖励设置，使得合作策略同时也对应智能体的自利策略。与现有学习激励自利智能体的方法不同，IQ-Flow不假设智能体的策略或学习算法，从而使得所开发框架能泛化至更广泛的应用场景。IQ-Flow利用其他智能体提供的数据离线评估学习策略的最优性，以确定合作策略与自利策略。随后采用元梯度学习估计策略评估如何随给定激励而变化，并修改激励使得合作目标与自利目标下的贪婪策略产生相同动作。我们在迭代矩阵博弈中展示了IQ-Flow的运行特性。实验证明，在逃生室和双人清洁环境中，IQ-Flow的表现优于最先进的激励设计算法。此外，我们进一步证明，预训练的IQ-Flow机制在双人清洁环境中的表现显著优于共享奖励设置。