Video dubbing has broad applications in filmmaking, multimedia creation, and assistive speech technology. Existing approaches either train directly on limited dubbing datasets or adopt a two-stage pipeline that adapts pre-trained text-to-speech (TTS) models, which often struggle to produce expressive prosody, rich acoustic characteristics, and precise synchronization. To address these issues, we propose DiFlowDubber with a novel two-stage training framework that effectively transfers knowledge from a pre-trained TTS model to video-driven dubbing, with a discrete flow matching generative backbone. Specifically, we design a FaPro module that captures global prosody and stylistic cues from facial expressions and leverages this information to guide the modeling of subsequent speech attributes. To ensure precise speech-lip synchronization, we introduce a Synchronizer module that bridges the modality gap among text, video, and speech, thereby improving cross-modal alignment and generating speech that is temporally synchronized with lip movements. Experiments on two primary benchmark datasets demonstrate that DiFlowDubber outperforms previous methods across multiple metrics.

翻译：视频配音在电影制作、多媒体创作和辅助语音技术中具有广泛应用。现有方法要么直接使用有限的配音数据集进行训练，要么采用两阶段流水线适配预训练的文本转语音模型，但这些方法通常难以生成富有表现力的韵律、丰富的声学特征以及精确的同步效果。为解决这些问题，我们提出DiFlowDubber，其采用包含离散流匹配生成骨干网络的新型两阶段训练框架，有效将预训练文本转语音模型的知识迁移至视频驱动配音任务。具体而言，我们设计了FaPro模块，用于从面部表情中捕获全局韵律和风格线索，并利用该信息指导后续语音属性的建模。为确保精确的语音-唇形同步，我们引入了同步器模块，该模块弥合了文本、视频和语音之间的模态差距，从而提升跨模态对齐效果，并生成与唇部运动时间同步的语音。在两个主要基准数据集上的实验表明，DiFlowDubber在多项指标上均优于以往方法。