Direct Preference Optimization (DPO) has recently improved Text-to-Video (T2V) generation by enhancing visual fidelity and text alignment. However, current methods rely on non-differentiable preference signals from human annotations or learned reward models. This reliance makes training label-intensive, bias-prone, and easy-to-game, which often triggers reward hacking and unstable training. We propose Diffusion-DRF, a differentiable reward flow for fine-tuning video diffusion models using a frozen, off-the-shelf Vision-Language Model (VLM) as a training-free critic. Diffusion-DRF directly backpropagates VLM feedback through the diffusion denoising chain, converting logit-level responses into token-aware gradients for optimization. We propose an automated, aspect-structured prompting pipeline to obtain reliable multi-dimensional VLM feedback, while gradient checkpointing enables efficient updates through the final denoising steps. Diffusion-DRF improves video quality and semantic alignment while mitigating reward hacking and collapse -- without additional reward models or preference datasets. It is model-agnostic and readily generalizes to other diffusion-based generative tasks.

翻译：直接偏好优化（DPO）近期通过提升视觉保真度与文本对齐改进了文本到视频（T2V）生成。然而，现有方法依赖于来自人工标注或学习型奖励模型的不可微分偏好信号。这种依赖性使得训练需要大量标注、易产生偏差且易于被操纵，常导致奖励破解与训练不稳定。我们提出Diffusion-DRF，一种用于视频扩散模型微分的可微分奖励流，其采用冻结的现成视觉语言模型（VLM）作为免训练评判器。Diffusion-DRF通过扩散去噪链直接反向传播VLM反馈，将逻辑层面的响应转化为面向令牌的梯度以进行优化。我们提出一种自动化、按维度结构化的提示流程，以获取可靠的多维度VLM反馈，同时梯度检查点技术实现了通过最终去噪步骤的高效更新。Diffusion-DRF在提升视频质量与语义对齐的同时，有效缓解了奖励破解与崩溃问题——无需额外的奖励模型或偏好数据集。该方法具有模型无关性，可轻松推广至其他基于扩散的生成任务。