Diffusion models have recently shown strong potential in language modeling, offering faster generation compared to traditional autoregressive approaches. However, applying supervised fine-tuning (SFT) to diffusion models remains challenging, as they lack precise probability estimates at each denoising step. While the diffusion mechanism enables the model to reason over entire sequences, it also makes the generation process less predictable and often inconsistent. This highlights the importance of controlling key tokens that guide the direction of generation. To address this issue, we propose GIFT, an importance-aware finetuning method for diffusion language models, where tokens are assigned different importance weights based on their entropy. Derived from diffusion theory, GIFT delivers substantial gains: across diverse settings including different mainstream training datasets ranging from 1k to 10k in size, utilizing LoRA or full parameter fine-tuning, and training on base or instruct models, GIFT consistently achieves superior overall performance compared to standard SFT on four widely used reasoning benchmarks (Sudoku, Countdown, GSM8K, and MATH-500).

翻译：扩散模型近期在语言建模中展现出强大潜力，相比传统自回归方法能实现更快的生成速度。然而，将监督式微调（SFT）应用于扩散模型仍面临挑战，因为这类模型在每个去噪步骤中缺乏精确的概率估计。扩散机制虽使模型能够对完整序列进行推理，但也导致生成过程难以预测且常出现不一致性。这凸显出控制引导生成方向的关键词元的重要性。为解决这一问题，我们提出GIFT——一种面向扩散语言模型的重要性感知微调方法，该方法根据词元熵值为其分配不同的重要性权重。基于扩散理论推导的GIFT带来显著性能提升：在多样化设置下——包括使用1k至10k规模的不同主流训练数据集、采用LoRA或全参数微调、基于基础模型或指令模型进行训练——相比标准SFT方法，GIFT在四项广泛使用的推理基准（数独、倒计时、GSM8K和MATH-500）上均持续取得更优的整体性能。