Timely delivery of delay-sensitive information over dynamic, heterogeneous networks is increasingly essential for a range of interactive applications, such as industrial automation, self-driving vehicles, and augmented reality. However, most existing network control solutions target only average delay performance, falling short of providing strict End-to-End (E2E) peak latency guarantees. This paper addresses the challenge of reliably delivering packets within application-imposed deadlines by leveraging recent advancements in Multi-Agent Deep Reinforcement Learning (MA-DRL). After introducing the Delay-Constrained Maximum-Throughput (DCMT) dynamic network control problem, and highlighting the limitations of current solutions, we present a novel MA-DRL network control framework that leverages a centralized routing and distributed scheduling architecture. The proposed framework leverages critical networking domain knowledge for the design of effective MA-DRL strategies based on the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) technique, where centralized routing and distributed scheduling agents dynamically assign paths and schedule packet transmissions according to packet lifetimes, thereby maximizing on-time packet delivery. The generality of the proposed framework allows integrating both data-driven \blue{Deep Reinforcement Learning (DRL)} agents and traditional rule-based policies in order to strike the right balance between performance and learning complexity. Our results confirm the superiority of the proposed framework with respect to traditional stochastic optimization-based approaches and provide key insights into the role and interplay between data-driven DRL agents and new rule-based policies for both efficient and high-performance control of latency-critical services.

翻译：在动态异构网络中及时传输延迟敏感信息对于工业自动化、自动驾驶汽车和增强现实等一系列交互式应用日益重要。然而，现有网络控制方案大多仅针对平均延迟性能进行优化，无法提供严格的端到端峰值延迟保障。本文通过利用多智能体深度强化学习的最新进展，解决了在应用规定时限内可靠传输数据包的挑战。在引入延迟约束最大吞吐量动态网络控制问题并指出当前解决方案的局限性后，我们提出了一种新颖的多智能体深度强化学习网络控制框架，该框架采用集中式路由与分布式调度架构。所提出的框架利用关键网络领域知识，基于多智能体深度确定性策略梯度技术设计了有效的多智能体深度强化学习策略，其中集中式路由与分布式调度智能体根据数据包生存期动态分配路径并调度数据包传输，从而最大化准时数据包交付率。该框架的通用性允许同时集成数据驱动的深度强化学习智能体与传统基于规则的策略，以在性能与学习复杂度之间取得最佳平衡。实验结果证实了所提框架相对于传统基于随机优化的方法的优越性，并为数据驱动的深度强化学习智能体与新型基于规则的策略在延迟关键型服务的高效高性能控制中的作用与相互作用提供了关键见解。