Diffusion language models (DLMs) enable parallel, order-agnostic generation with iterative refinement, offering a flexible alternative to autoregressive large language models (LLMs). However, adapting reinforcement learning (RL) fine-tuning to DLMs remains an open challenge because of the intractable likelihood. Pioneering work such as diffu-GRPO estimated token-level likelihoods via one-step unmasking. While computationally efficient, this approach is severely biased. A more principled foundation lies in sequence-level likelihoods, where the evidence lower bound (ELBO) serves as a surrogate. Yet, despite this clean mathematical connection, ELBO-based methods have seen limited adoption due to the prohibitive cost of likelihood evaluation. In this work, we revisit ELBO estimation and disentangle its sources of variance. This decomposition motivates reducing variance through fast, deterministic integral approximations along a few pivotal dimensions. Building on this insight, we introduce Group Diffusion Policy Optimization (GDPO), a new RL algorithm tailored for DLMs. GDPO leverages simple yet effective Semi-deterministic Monte Carlo schemes to mitigate the variance explosion of ELBO estimators under vanilla double Monte Carlo sampling, yielding a provably lower-variance estimator under tight evaluation budgets. Empirically, GDPO achieves consistent gains over pretrained checkpoints and outperforms diffu-GRPO, one of the state-of-the-art baselines, on the majority of math, reasoning, and coding benchmarks.

翻译：扩散语言模型（DLMs）通过并行、顺序无关的生成与迭代优化，为自回归大语言模型（LLMs）提供了灵活的替代方案。然而，由于似然函数的难解性，将强化学习（RL）微调应用于DLMs仍是一个开放挑战。开创性工作如diffu-GRPO通过单步解掩码估计了词元级似然。该方法虽计算高效，但存在严重偏差。更理论完备的基础在于序列级似然，其中证据下界（ELBO）可作为替代指标。然而，尽管存在清晰的数学关联，基于ELBO的方法因似然评估的过高计算成本而应用有限。本研究重新审视ELBO估计方法，并解构其方差来源。该分解表明可通过沿少数关键维度进行快速、确定的积分近似来降低方差。基于此洞见，我们提出了分组扩散策略优化（GDPO），一种专为DLMs设计的新型RL算法。GDPO利用简洁高效的半确定性蒙特卡洛方案，缓解了在朴素双重蒙特卡洛采样下ELBO估计器的方差爆炸问题，从而在严格评估预算下构建了理论上更低方差的估计器。实证结果表明，GDPO在多数数学、推理和代码生成基准测试中，相较于预训练检查点取得持续提升，并超越了当前先进基线方法之一的diffu-GRPO。