Effectively scaling Reinforcement Learning (RL) is crucial for enhancing the reasoning and alignment of Large Language Models. The massive data and complex execution flows inherent in these tasks require a distributed architecture capable of efficient scaling. However, to simplify programming and dependency management, mainstream frameworks often rely on a centralized architecture where a single node dispatches both control and data. This inherent coupling creates significant communication bottlenecks, severely limiting system scalability and efficiency. We present DISTFLOW, a novel, fully distributed RL framework that adopts a multi-controller paradigm. By decoupling data transmission from control dispatch, DISTFLOW establishes a parallelism-aware, decentralized Data Coordinator that leverages local caching, load balancing, and asynchronous double buffer to minimize communication overhead and mitigate straggler effects. For control logic, it introduces a task scheduler built upon Directed Acyclic Graph (DAG) that facilitates fine-grained, independent execution. Experimental results demonstrate that DISTFLOW achieves near-linear scalability up to 512 GPUs and delivers up to a 2.63x throughput improvement over state-of-the-art (SOTA) frameworks. The source code is available at: https://github.com/sii-research/siiRL.

翻译：有效扩展强化学习（RL）对于增强大型语言模型的推理能力与对齐效果至关重要。这些任务中蕴含的海量数据与复杂执行流程，需要一种能够高效扩展的分布式架构。然而，为简化编程和依赖管理，主流框架通常依赖集中式架构——由单一节点同时调度控制与数据。这种固有关联性造成了显著的通信瓶颈，严重限制了系统的可扩展性与效率。我们提出了DISTFLOW，一种全新的完全分布式RL框架，采用多控制器范式。通过将数据传输与控制调度解耦，DISTFLOW构建了一种具备并行感知能力的去中心化数据协调器，该协调器利用本地缓存、负载均衡与异步双缓冲机制，最小化通信开销并缓解掉队效应。对于控制逻辑，它引入了一种基于有向无环图（DAG）的任务调度器，支持细粒度的独立执行。实验结果表明，DISTFLOW在512块GPU上实现了近乎线性的可扩展性，与当前最优（SOTA）框架相比，吞吐量提升高达2.63倍。源代码已发布于：https://github.com/sii-research/siiRL。