Discrete flow models (DFMs) are a class of flexible generative models for generating discrete data, and diffusion large language models (dLLMs) can be viewed as a special case with a specific choice of mixture path and a masked source distribution. While several recent works have explored reinforcement learning into dLLMs, its application to more general discrete flow models remains underexplored. In this work, we present discrete Flow-GRPO (dFlowGRPO), a unified reinforcement learning framework for discrete flow models that supports a broad family of probability paths and non-masked source distributions. We derive the full trajectory probability for DFMs and formulate denoising as a Markov decision process, enabling dFlowGRPO to incorporate information from both the associated conditional transition rates and the posterior model during reinforcement learning. We apply dFlowGRPO to FUDOKI, a recent multimodal discrete flow model, and evaluate it on both image generation and multimodal understanding tasks. Empirical results show that dFlowGRPO outperforms existing GRPO-type methods for dLLMs on text-to-image generation tasks and achieves performance competitive with continuous flow-based models trained using FlowGRPO, while also demonstrating strong capabilities on understanding tasks.

翻译：离散流模型（DFM）是一类用于生成离散数据的灵活生成模型，扩散大语言模型（dLLM）可视为其特例，由特定的混合路径选择与掩码源分布构成。尽管近期多项研究已将强化学习引入dLLM，但该方法在更广义离散流模型中的应用仍鲜有探索。本文提出离散Flow-GRPO（dFlowGRPO）框架——一个支持概率路径泛化族与非掩码源分布的离散流模型统一强化学习框架。我们推导了DFM的完整轨迹概率，并将去噪过程建模为马尔可夫决策过程，使dFlowGRPO在强化学习过程中能同时利用关联条件转移率与后验模型的信息。将该方法应用于多模态离散流模型FUDOKI后，我们在图像生成与多模态理解任务上进行了评估。实验结果表明：在文本到图像生成任务中，dFlowGRPO性能优于现有面向dLLM的GRPO类方法；在与基于FlowGRPO训练的连续流模型对比时亦展现出相当竞争力，同时在理解任务中表现出色。