World models simulate environmental dynamics to enable agents to plan and reason about future states. While existing approaches have primarily focused on visual observations, real-world perception inherently involves multiple sensory modalities. Audio provides crucial spatial and temporal cues such as sound source localization and acoustic scene properties, yet its integration into world models remains largely unexplored. No prior work has formally defined what constitutes an audio-visual world model or how to jointly capture binaural spatial audio and visual dynamics under precise action control. This work presents the first formal framework for Audio-Visual World Models (AVWM), formulating multimodal environment simulation as a partially observable Markov decision process with synchronized audio-visual observations. To address the lack of suitable training data, we construct AVW-4k, a dataset comprising 30 hours of binaural audio-visual trajectories with action annotations across 76 indoor environments. We propose AV-CDiT, an Audio-Visual Conditional Diffusion Transformer with a novel modality expert architecture that balances visual and auditory learning, optimized through a three-stage training strategy for effective multimodal integration. Extensive experiments demonstrate that AV-CDiT achieves high-fidelity multimodal prediction across visual and auditory modalities. Furthermore, we validate its practical utility in continuous audio-visual navigation tasks, where AVWM significantly enhances the agent's performance.

翻译：世界模型通过模拟环境动态，使智能体能够对未来状态进行规划与推理。现有方法主要集中于视觉观测，而真实世界的感知本质上涉及多种感官模态。音频提供了关键的空间与时间线索，如声源定位与声学场景特性，但其在世界模型中的整合仍鲜有探索。此前尚无研究正式定义何谓视听世界模型，亦未阐明如何在精确动作控制下联合捕捉双耳空间音频与视觉动态。本研究首次提出了视听世界模型（AVWM）的形式化框架，将多模态环境模拟表述为具有同步视听观测的部分可观测马尔可夫决策过程。针对训练数据匮乏的问题，我们构建了AVW-4k数据集，包含76个室内环境中30小时带动作标注的双耳视听轨迹。我们提出AV-CDiT——一种采用新型模态专家架构的视听条件扩散Transformer，该架构通过三阶段训练策略优化，平衡视觉与听觉学习以实现有效的多模态整合。大量实验表明，AV-CDiT在视觉与听觉模态上均实现了高保真度的多模态预测。此外，我们在连续视听导航任务中验证了其实际效用，证明AVWM能显著提升智能体的性能。