Audio-driven visual dubbing aims to synchronize a video's lip movements with new speech, but is fundamentally challenged by the lack of ideal training data: paired videos where only a subject's lip movements differ while all other visual conditions are identical. Existing methods circumvent this with a mask-based inpainting paradigm, where an incomplete visual conditioning forces models to simultaneously hallucinate missing content and sync lips, leading to visual artifacts, identity drift, and poor synchronization. In this work, we propose a novel self-bootstrapping framework that reframes visual dubbing from an ill-posed inpainting task into a well-conditioned video-to-video editing problem. Our approach employs a Diffusion Transformer, first as a data generator, to synthesize ideal training data: a lip-altered companion video for each real sample, forming visually aligned video pairs. A DiT-based audio-driven editor is then trained on these pairs end-to-end, leveraging the complete and aligned input video frames to focus solely on precise, audio-driven lip modifications. This complete, frame-aligned input conditioning forms a rich visual context for the editor, providing it with complete identity cues, scene interactions, and continuous spatiotemporal dynamics. Leveraging this rich context fundamentally enables our method to achieve highly accurate lip sync, faithful identity preservation, and exceptional robustness against challenging in-the-wild scenarios. We further introduce a timestep-adaptive multi-phase learning strategy as a necessary component to disentangle conflicting editing objectives across diffusion timesteps, thereby facilitating stable training and yielding enhanced lip synchronization and visual fidelity. Additionally, we propose ContextDubBench, a comprehensive benchmark dataset for robust evaluation in diverse and challenging practical application scenarios.

翻译：音频驱动的视觉配音旨在将视频的唇部运动与新的语音同步，但根本上受到理想训练数据缺失的挑战：即缺乏仅主体唇部运动不同而所有其他视觉条件完全相同的配对视频。现有方法通过基于掩码的修复范式规避此问题，其中不完整的视觉条件迫使模型同时幻觉缺失内容并同步唇部，导致视觉伪影、身份漂移和同步效果差。在本工作中，我们提出了一种新颖的自引导框架，将视觉配音从一个不适定的修复任务重新构建为一个良条件的视频到视频编辑问题。我们的方法采用一个Diffusion Transformer，首先作为数据生成器，合成理想的训练数据：为每个真实样本生成一个唇部改变的伴生视频，形成视觉对齐的视频对。随后，一个基于DiT的音频驱动编辑器在这些视频对上端到端地进行训练，利用完整且对齐的输入视频帧，专注于精确的、音频驱动的唇部修改。这种完整、帧对齐的输入条件为编辑器提供了丰富的视觉上下文，赋予其完整的身份线索、场景交互和连续的时空动态。利用这种丰富上下文从根本上使我们的方法能够实现高度准确的唇部同步、忠实的身份保持，以及对具有挑战性的野外场景的卓越鲁棒性。我们进一步引入了一种时间步自适应的多阶段学习策略作为必要组件，以解耦扩散时间步之间冲突的编辑目标，从而促进稳定训练并提升唇部同步和视觉保真度。此外，我们提出了ContextDubBench，一个用于在多样且具有挑战性的实际应用场景中进行鲁棒评估的综合基准数据集。