Distributed Reinforcement Learning (RL) frameworks are essential for mapping RL workloads to multiple computational resources, allowing for faster generation of samples, estimation of values, and policy improvement. These computational paradigms require a seamless integration of training, serving, and simulation workloads. Existing frameworks, such as Ray, are not managing this orchestration efficiently. In this study, we've proposed a solution implementing Reactor Model, which enforces a set of actors to have a fixed communication pattern. This allows the scheduler to eliminate works needed for synchronization, such as acquiring and releasing locks for each actor or sending and processing coordination-related messages. Our framework, Lingua Franca (LF), a coordination language based on the Reactor Model, also provides a unified interface that allows users to automatically generate dataflow graphs for distributed RL. On average, LF outperformed Ray in generating samples from OpenAI Gym and Atari environments by 1.21x and 11.62x, reduced the average training time of synchronized parallel Q-learning by 31.2%, and accelerated Multi-Agent RL inference by 5.12x.

翻译：分布式强化学习框架对于将强化学习任务映射到多计算资源至关重要，可加速样本生成、价值估计及策略改进。此类计算范式要求训练、推理与仿真工作负载的无缝集成。现有框架（如Ray）在编排管理方面效率不足。本研究提出基于Reactor模型的解决方案，通过强制规定一组参与者采用固定通信模式，使得调度器能够消除同步操作（如为每个参与者获取/释放锁、发送及处理协调相关消息）所需的计算开销。我们的框架Lingua Franca（LF）作为基于Reactor模型的协调语言，提供统一接口，支持用户自动生成分布式强化学习的数据流图。实验表明，在OpenAI Gym和Atari环境中的样本生成速度上，LF平均比Ray提升1.21倍和11.62倍；同步并行Q-learning的平均训练时间减少31.2%；多智能体强化学习推理加速比达5.12倍。