In recent years, Vision-Language-Action (VLA) models have emerged as a crucial pathway towards general embodied intelligence, yet their training efficiency has become a key bottleneck. Although existing reinforcement learning (RL)-based training frameworks like RLinf can enhance model generalization, they still rely on synchronous execution, leading to severe resource underutilization and throughput limitations during environment interaction, policy generation (rollout), and model update phases (actor). To overcome this challenge, this paper, for the first time, proposes and implements a fully-asynchronous policy training framework encompassing the entire pipeline from environment interaction, rollout generation, to actor policy updates. Systematically drawing inspiration from asynchronous optimization ideas in large model RL, our framework designs a multi-level decoupled architecture. This includes asynchronous parallelization of environment interaction and trajectory collection, streaming execution for policy generation, and decoupled scheduling for training updates. We validated the effectiveness of our method across diverse VLA models and environments. On the LIBERO benchmark, the framework achieves throughput improvements of up to 59.25\% compared to existing synchronous strategies. When deeply optimizing separation strategies, throughput can be increased by as much as 126.67\%. We verified the effectiveness of each asynchronous component via ablation studies. Scaling law validation across 8 to 256 GPUs demonstrates our method's excellent scalability under most conditions.

翻译：近年来，视觉-语言-动作（VLA）模型已成为实现通用具身智能的关键路径，但其训练效率已成为主要瓶颈。尽管现有的基于强化学习（RL）的训练框架（如RLinf）能够提升模型泛化能力，但它们仍依赖于同步执行，导致在环境交互、策略生成（rollout）和模型更新（actor）阶段存在严重的资源利用不足与吞吐量限制。为克服这一挑战，本文首次提出并实现了一个完全异步的策略训练框架，该框架涵盖了从环境交互、rollout生成到actor策略更新的全流程。系统性地借鉴大模型RL中的异步优化思想，我们的框架设计了一种多层次解耦架构。这包括环境交互与轨迹收集的异步并行化、策略生成的流式执行以及训练更新的解耦调度。我们在多种VLA模型与环境上验证了方法的有效性。在LIBERO基准测试中，该框架相较于现有同步策略实现了高达59.25\%的吞吐量提升。在深度优化分离策略时，吞吐量最高可提升126.67\%。我们通过消融实验验证了各异步组件的有效性。在8至256个GPU上的扩展律验证表明，我们的方法在多数条件下具有优异的可扩展性。