Multimodal generative models have shown remarkable progress in single-modality video and audio synthesis, yet truly joint audio-video generation remains an open challenge. In this paper, I explore four key contributions to advance this field. First, I release two high-quality, paired audio-video datasets. The datasets consisting on 13 hours of video-game clips and 64 hours of concert performances, each segmented into consistent 34-second samples to facilitate reproducible research. Second, I train the MM-Diffusion architecture from scratch on our datasets, demonstrating its ability to produce semantically coherent audio-video pairs and quantitatively evaluating alignment on rapid actions and musical cues. Third, I investigate joint latent diffusion by leveraging pretrained video and audio encoder-decoders, uncovering challenges and inconsistencies in the multimodal decoding stage. Finally, I propose a sequential two-step text-to-audio-video generation pipeline: first generating video, then conditioning on both the video output and the original prompt to synthesize temporally synchronized audio. My experiments show that this modular approach yields high-fidelity generations of audio video generation.

翻译：多模态生成模型在单模态视频与音频合成领域取得了显著进展，然而真正意义上的联合音频-视频生成仍是一项开放挑战。本文通过四项关键贡献推动该领域发展：首先，公开两个高质量配对音频-视频数据集，包含13小时游戏剪辑与64小时音乐会表演片段，每段分割为一致的34秒样本以促进可重复研究；其次，在这些数据集上从头训练MM-Diffusion架构，证明其生成语义连贯音视频对的能力，并通过快速动作与音乐线索的对齐性进行定量评估；第三，利用预训练的视频与音频编码-解码器探索联合潜在扩散，揭示多模态解码阶段存在的挑战与不一致性；最后提出一种顺序式两步文本到音频-视频生成流水线：先生成视频，再以视频输出与原始提示为条件合成时间同步的音频。实验表明，这种模块化方法能够生成高保真度的音频-视频内容。