We study reinforcement learning for global decision-making in the presence of many local agents, where the global decision-maker makes decisions affecting all local agents, and the objective is to learn a policy that maximizes the rewards of both the global and the local agents. Such problems find many applications, e.g. demand response, EV charging, queueing, etc. In this setting, scalability has been a long-standing challenge due to the size of the state/action space which can be exponential in the number of agents. This work proposes the SUB-SAMPLE-Q algorithm where the global agent subsamples $k\leq n$ local agents to compute an optimal policy in time that is only exponential in $k$, providing an exponential speedup from standard methods that are exponential in $n$. We show that the learned policy converges to the optimal policy in the order of $\tilde{O}(1/\sqrt{k}+\epsilon_{k,m})$ as the number of sub-sampled agents $k$ increases, where $\epsilon_{k,m}$ is the Bellman noise. We also conduct numerical simulations in a demand-response setting and a queueing setting.

翻译：我们研究了在众多局部智能体环境下进行全局决策的强化学习问题。在此场景中，全局决策者制定的决策影响所有局部智能体，其目标在于学习一种能够同时最大化全局与局部智能体收益的策略。此类问题在需求响应、电动汽车充电、排队等领域具有广泛应用。由于状态/动作空间维度可能随智能体数量呈指数级增长，可扩展性一直是该领域长期存在的挑战。本文提出SUB-SAMPLE-Q算法，全局智能体通过子采样k≤n个局部智能体，以仅对k呈指数级的时间复杂度计算最优策略，相比对n呈指数级的标准方法实现指数级加速。我们证明，随着子采样智能体数量k的增加，学得策略以$\tilde{O}(1/\sqrt{k}+\epsilon_{k,m})$的阶收敛于最优策略，其中$\epsilon_{k,m}$为贝尔曼噪声项。此外，我们在需求响应与排队场景中进行了数值仿真验证。