Recent advances in Diffusion Transformers (DiTs) have enabled high-quality joint audio-video generation, producing videos with synchronized audio within a single model. However, existing controllable generation frameworks are typically restricted to video-only control. This restricts comprehensive controllability and often leads to suboptimal cross-modal alignment. To bridge this gap, we present MMControl, which enables users to perform Multi-Modal Control in joint audio-video generation. MMControl introduces a dual-stream conditional injection mechanism. It incorporates both visual and acoustic control signals, including reference images, reference audio, depth maps, and pose sequences, into a joint generation process. These conditions are injected through bypass branches into a joint audio-video Diffusion Transformer, enabling the model to simultaneously generate identity-consistent video and timbre-consistent audio under structural constraints. Furthermore, we introduce modality-specific guidance scaling, which allows users to independently and dynamically adjust the influence strength of each visual and acoustic condition at inference time. Extensive experiments demonstrate that MMControl achieves fine-grained, composable control over character identity, voice timbre, body pose, and scene layout in joint audio-video generation.

翻译：近年来，扩散变换器（Diffusion Transformers, DiTs）的进展使得高质量音视频联合生成成为可能，能够在单一模型内生成与音频同步的视频。然而，现有的可控生成框架通常局限于视频单模态控制，这限制了全面的可控性，并常导致跨模态对齐效果欠佳。为弥补这一差距，我们提出MMControl，它能够在音视频联合生成中实现多模态控制。MMControl引入了一种双流条件注入机制，将视觉与声学控制信号（包括参考图像、参考音频、深度图与姿态序列）整合至联合生成过程中。这些条件通过旁路分支注入到音视频联合扩散变换器中，使得模型能够在结构约束下同步生成身份一致的视频与音色一致的音频。此外，我们引入了模态特定的引导缩放机制，允许用户在推理时独立且动态地调整每种视觉与声学条件的影响强度。大量实验表明，MMControl能够在音视频联合生成中实现对角色身份、嗓音音色、身体姿态及场景布局的细粒度、可组合控制。