Recent advances in video generation have enabled thrilling experiences in producing realistic videos driven by scalable diffusion transformers. However, they usually fail to produce satisfactory outputs that are aligned to users' authentic demands and preferences. In this work, we introduce Dual-Iterative Optimization (Dual-IPO), an iterative paradigm that sequentially optimizes both the reward model and the video generation model for improved synthesis quality and human preference alignment. For the reward model, our framework ensures reliable and robust reward signals via CoT-guided reasoning, voting-based self-consistency, and preference certainty estimation. Given this, we optimize video foundation models with guidance of signals from reward model's feedback, thus improving the synthesis quality in subject consistency, motion smoothness and aesthetic quality, etc. The reward model and video generation model complement each other and are progressively improved in the multi-round iteration, without requiring tediously manual preference annotations. Comprehensive experiments demonstrate that the proposed Dual-IPO can effectively and consistently improve the video generation quality of base model with various architectures and sizes, even help a model with only 2B parameters surpass a 5B one. Moreover, our analysis experiments and ablation studies identify the rational of our systematic design and the efficacy of each component.

翻译：近年来，视频生成领域取得了显著进展，基于可扩展扩散Transformer的方法已能生成逼真的视频内容。然而，现有方法通常难以生成符合用户真实需求与偏好的满意输出。本研究提出双迭代优化（Dual-IPO），这是一种通过顺序优化奖励模型与视频生成模型以提升合成质量与人类偏好对齐的迭代范式。对于奖励模型，我们通过思维链引导推理、基于投票的自洽性机制以及偏好确定性估计，确保奖励信号的可靠性与鲁棒性。在此基础上，我们利用奖励模型的反馈信号指导视频基础模型的优化，从而在主体一致性、运动平滑度与美学质量等方面提升合成质量。奖励模型与视频生成模型在无需繁琐人工标注的情况下，通过多轮迭代相互促进并持续改进。综合实验表明，所提出的Dual-IPO方法能有效且持续地提升不同架构与规模基础模型的视频生成质量，甚至能使仅含20亿参数的模型性能超越50亿参数模型。此外，我们的分析实验与消融研究验证了系统设计的合理性及各组件的有效性。