Dynamic Population Games (DPGs) provide a tractable framework for modeling strategic interactions in large populations of self-interested agents, and have been successfully applied to the design of Karma economies, a class of fair non-monetary resource allocation mechanisms. Despite their appealing theoretical properties, existing computational tools for DPGs assume full knowledge of the game model and operate in a centralized fashion, limiting their applicability in realistic settings where agents have access only to their own private experience. This paper takes a step towards addressing this gap by studying model-free equilibrium learning in Karma DPGs. First, we analyze the setting in which a novel agent joins a Karma DPG already at its Stationary Nash Equilibrium (SNE) and learns a policy via Deep Q-Networks (DQN) without knowledge of the game model. Leveraging recent convergence results for DQN, we establish a suboptimality bound consisting of a DQN approximation error of order $O(1/\sqrt{N_s})$ and a mean field perturbation error of order $O(1/N)$, where $N_s$ is the replay buffer size and $N$ is the population size. Second, we consider the challenging problem of learning the SNE from scratch. We show empirically that combining deep RL with fictitious play and smoothed policy iteration allows agents to converge, in a model-free fashion, to a configuration close to the centrally computed SNE. Together, these contributions support the vision of Karma economies as practical tools for fair resource allocation.

翻译：摘要：动态群体博弈（DPGs）为建模大规模自利主体间的策略交互提供了易处理的框架，并已成功应用于因果经济（Karma economies）设计——一类公平的非货币资源分配机制。尽管其理论性质极具吸引力，现有DPG计算方法均假设完全掌握博弈模型并以集中式方式运行，这限制了在主体仅能获取自身私有经验的实际场景中的适用性。本文通过研究因果DPG中的无模型均衡学习，向填补这一空白迈出关键一步。首先，我们分析新主体加入已处于平稳纳什均衡（SNE）的因果DPG并利用深度Q网络（DQN）学习策略的情形，该过程无需知晓博弈模型。借助DQN最新收敛性结论，我们建立了次优性界：包含阶数为$O(1/\sqrt{N_s})$的DQN近似误差与阶数为$O(1/N)$的均场扰动误差，其中$N_s$为回放缓冲区大小，$N$为种群规模。其次，我们考虑从零学习SNE这一具有挑战性的问题。实证结果表明，将深度强化学习与虚拟博弈及平滑策略迭代相结合，可使主体以无模型方式收敛至接近集中式计算出的SNE配置。这些成果共同支持了因果经济作为公平资源分配实用工具的愿景。