The intrinsic link between facial motion and speech is often overlooked in generative modeling, where talking head synthesis and text-to-speech (TTS) are typically addressed as separate tasks. This paper introduces JAM-Flow, a unified framework to simultaneously synthesize and condition on both facial motion and speech. Our approach leverages flow matching and a novel Multi-Modal Diffusion Transformer (MM-DiT) architecture, integrating specialized Motion-DiT and Audio-DiT modules. These are coupled via selective joint attention layers and incorporate key architectural choices, such as temporally aligned positional embeddings and localized joint attention masking, to enable effective cross-modal interaction while preserving modality-specific strengths. Trained with an inpainting-style objective, JAM-Flow supports a wide array of conditioning inputs-including text, reference audio, and reference motion-facilitating tasks such as synchronized talking head generation from text, audio-driven animation, and much more, within a single, coherent model. JAM-Flow significantly advances multi-modal generative modeling by providing a practical solution for holistic audio-visual synthesis. project page: https://joonghyuk.com/jamflow-web

翻译：面部运动与语音之间的内在联系在生成建模中常被忽视，说话头像合成与文本转语音（TTS）通常被视为独立任务。本文提出JAM-Flow，一个统一框架，可同时合成并条件化处理面部运动与语音。该方法采用流匹配技术与新型多模态扩散Transformer（MM-DiT）架构，集成专用运动扩散Transformer（Motion-DiT）与音频扩散Transformer（Audio-DiT）模块。这些模块通过选择性联合注意力层耦合，并融入关键架构设计（如时间对齐的位置编码与局部联合注意力掩码），在保留模态特定优势的同时实现高效跨模态交互。JAM-Flow采用修复式训练目标，支持文本、参考音频、参考运动等多种条件输入，从而在单一连贯模型中实现文本驱动的同步说话头像生成、音频驱动动画等丰富任务。该模型通过提供整体音视频合成的实用方案，显著推进了多模态生成建模的发展。项目主页：https://joonghyuk.com/jamflow-web