Although GRPO substantially enhances flow matching models in human preference alignment of image generation, methods such as FlowGRPO and DanceGRPO still exhibit inefficiency due to the necessity of sampling and optimizing over all denoising steps specified by the Markov Decision Process (MDP). In this paper, we propose $\textbf{MixGRPO}$, a novel framework that leverages the flexibility of mixed sampling strategies through the integration of stochastic differential equations (SDE) and ordinary differential equations (ODE). This streamlines the optimization process within the MDP to improve efficiency and boost performance. Specifically, MixGRPO introduces a sliding window mechanism, using SDE sampling and GRPO-guided optimization only within the window, while applying ODE sampling outside. This design confines sampling randomness to the time-steps within the window, thereby reducing the optimization overhead, and allowing for more focused gradient updates to accelerate convergence. Additionally, as time-steps beyond the sliding window are not involved in optimization, higher-order solvers are supported for faster sampling. So we present a faster variant, termed $\textbf{MixGRPO-Flash}$, which further improves training efficiency while achieving comparable performance. MixGRPO exhibits substantial gains across multiple dimensions of human preference alignment, outperforming DanceGRPO in both effectiveness and efficiency, with nearly 50% lower training time. Notably, MixGRPO-Flash further reduces training time by 71%.

翻译：尽管GRPO在图像生成的人类偏好对齐中显著增强了流匹配模型，但FlowGRPO和DanceGRPO等方法仍因需对马尔可夫决策过程（MDP）指定的所有去噪步骤进行采样和优化而表现出低效性。本文提出$\textbf{MixGRPO}$，一种通过整合随机微分方程（SDE）和常微分方程（ODE）来利用混合采样策略灵活性的新颖框架。该框架简化了MDP内的优化过程，以提高效率并提升性能。具体而言，MixGRPO引入滑动窗口机制，仅在窗口内使用SDE采样和GRPO引导的优化，而在窗口外应用ODE采样。此设计将采样随机性限制在窗口内的时间步，从而减少优化开销，并允许更聚焦的梯度更新以加速收敛。此外，由于滑动窗口外的时间步不参与优化，支持使用高阶求解器以实现更快采样。因此，我们提出一个更快的变体，称为$\textbf{MixGRPO-Flash}$，它在实现相当性能的同时进一步提升了训练效率。MixGRPO在人类偏好对齐的多个维度上展现出显著优势，在效果和效率上均优于DanceGRPO，训练时间降低近50%。值得注意的是，MixGRPO-Flash进一步将训练时间减少了71%。