In this work, we propose Mutual Forcing, a framework for fast autoregressive audio-video generation with long-horizon audio-video synchronization. Our approach addresses two key challenges: joint audio-video modeling and fast autoregressive generation. To ease joint audio-video optimization, we adopt a two-stage training strategy: we first train uni-modal generators and then couple them into a unified audio-video model for joint training on paired data. For streaming generation, we ask whether a native fast causal audio-video model can be trained directly, instead of following existing streaming distillation pipelines that typically train a bidirectional model first and then convert it into a causal generator through multiple distillation stages. Our answer is Mutual Forcing, which builds directly on native autoregressive model and integrates few-step and multi-step generation within a single weight-shared model, enabling self-distillation and improved training-inference consistency. The multi-step mode improves the few-step mode via self-distillation, while the few-step mode generates historical context during training to improve training-inference consistency; because the two modes share parameters, these two effects reinforce each other within a single model. Compared with prior approaches such as Self-Forcing, Mutual Forcing removes the need for an additional bidirectional teacher model, supports more flexible training sequence lengths, reduces training overhead, and allows the model to improve directly from real paired data rather than a fixed teacher. Experiments show that Mutual Forcing matches or surpasses strong baselines that require around 50 sampling steps while using only 4 to 8 steps, demonstrating substantial advantages in both efficiency and quality. The project page is available at https://mutualforcing.github.io.
翻译:本文提出互强迫(Mutual Forcing)框架,用于实现长时音视频同步的快速自回归生成。该方法解决了两个关键挑战:联合音视频建模与快速自回归生成。为缓解联合优化难度,我们采用两阶段训练策略:首先训练单模态生成器,随后将其耦合成统一的音视频模型,在配对数据上进行联合训练。针对流式生成,我们探究是否可以直接训练原生快速因果音视频模型,而非遵循现有流式蒸馏流程(通常先训练双向模型,再经过多阶段蒸馏转化为因果生成器)。我们的答案是互强迫,该框架直接基于原生自回归模型,将少步和多步生成集成于单一共享权重的模型中,实现自蒸馏并提升训练-推理一致性:多步模式通过自蒸馏优化少步模式,而少步模式在训练期间生成历史上下文以增强训练-推理一致性——由于两种模式共享参数,这两个效应在单一模型内相互强化。相比自强迫(Self-Forcing)等先前方法,互强迫无需额外的双向教师模型,支持更灵活的训练序列长度,降低训练开销,并允许模型直接从真实配对数据而非固定教师模型中改进。实验表明,互强迫在仅使用4至8步采样的情况下,即可匹配甚至超越需要约50步采样的强基线方法,在效率和生成质量上均展现出显著优势。项目页面见https://mutualforcing.github.io。