The ever-growing complexity of reinforcement learning (RL) tasks demands a distributed system to efficiently generate and process a massive amount of data. However, existing open-source libraries suffer from various limitations, which impede their practical use in challenging scenarios where large-scale training is necessary. In this paper, we present a novel abstraction on the dataflows of RL training, which unifies diverse RL training applications into a general framework. Following this abstraction, we develop a scalable, efficient, and extensible distributed RL system called ReaLlyScalableRL, which allows efficient and massively parallelized training and easy development of customized algorithms. Our evaluation shows that SRL outperforms existing academic libraries, reaching at most 21x higher training throughput in a distributed setting. On learning performance, beyond performing and scaling well on common RL benchmarks with different RL algorithms, SRL can reproduce the same solution in the challenging hide-and-seek environment as reported by OpenAI with up to 5x speedup in wall-clock time. Notably, SRL is the first in the academic community to perform RL experiments at a large scale with over 15k CPU cores. SRL source code is available at: https://github.com/openpsi-project/srl .

翻译：强化学习（RL）任务的日益复杂性要求采用分布式系统来高效生成和处理海量数据。然而，现有的开源库存在诸多局限性，阻碍了其在需要大规模训练的复杂场景中的实际应用。本文提出了一种针对RL训练数据流的新型抽象，将多样化的RL训练应用统一到一个通用框架中。基于该抽象，我们开发了一个可扩展、高效且可扩展的分布式RL系统——ReaLlyScalableRL，该系统支持高效的大规模并行训练，并便于开发定制化算法。评估结果表明，SRL在分布式环境下优于现有学术库，最高可实现21倍的训练吞吐量提升。在学习性能方面，除了在不同RL算法的常见基准测试中表现优异且具有良好的扩展性外，SRL还能在OpenAI所报告的复杂“捉迷藏”环境中复现相同解决方案，且实际运行时间最高可加速5倍。值得注意的是，SRL是学术界首个在超过1.5万个CPU核心上开展大规模RL实验的系统。SRL源代码已公开于：https://github.com/openpsi-project/srl。