Multi-agent Deep Reinforcement Learning (MADRL) based traffic signal control becomes a popular research topic in recent years. To alleviate the scalability issue of completely centralized RL techniques and the non-stationarity issue of completely decentralized RL techniques on large-scale traffic networks, some literature utilizes a regional control approach where the whole network is firstly partitioned into multiple disjoint regions, followed by applying the centralized RL approach to each region. However, the existing partitioning rules either have no constraints on the topology of regions or require the same topology for all regions. Meanwhile, no existing regional control approach explores the performance of optimal joint action in an exponentially growing regional action space when intersections are controlled by 4-phase traffic signals (EW, EWL, NS, NSL). In this paper, we propose a novel RL training framework named RegionLight to tackle the above limitations. Specifically, the topology of regions is firstly constrained to a star network which comprises one center and an arbitrary number of leaves. Next, the network partitioning problem is modeled as an optimization problem to minimize the number of regions. Then, an Adaptive Branching Dueling Q-Network (ABDQ) model is proposed to decompose the regional control task into several joint signal control sub-tasks corresponding to particular intersections. Subsequently, these sub-tasks maximize the regional benefits cooperatively. Finally, the global control strategy for the whole network is obtained by concatenating the optimal joint actions of all regions. Experimental results demonstrate the superiority of our proposed framework over all baselines under both real and synthetic datasets in all evaluation metrics.

翻译：近年来，基于多智能体深度强化学习的交通信号控制成为热门研究课题。为缓解完全集中式强化学习技术在大规模交通网络中的可扩展性问题，以及完全分布式强化学习技术面临的非平稳性问题，部分研究采用区域控制方法：首先将整个网络划分为多个不相交区域，随后在每个区域内应用集中式强化学习。然而，现有分区规则要么对区域拓扑结构无约束，要么要求所有区域具有相同拓扑结构。同时，当交叉口采用4相位交通信号控制时，现有区域控制方法均未探索指数级增长的区域动作空间中最优联合动作的性能。本文提出一种名为RegionLight的新型强化学习训练框架以解决上述局限性。具体而言，首先将区域拓扑约束为由一个中心节点与任意数量叶节点组成的星型网络；其次将网络分区问题建模为以最小化区域数量为目标的最优化问题；随后提出自适应分支双Q网络模型，将区域控制任务分解为对应特定交叉口的若干联合信号控制子任务，这些子任务通过协同作用最大化区域效益；最后通过拼接所有区域的最优联合动作获得整个网络的全局控制策略。实验结果表明，在真实数据集与合成数据集的所有评估指标上，所提框架均优于所有基线方法。