The recent advancement of Artificial Intelligence Generated Content (AIGC) has led to significant strides in modeling human interaction, particularly in the context of multimodal dialogue. While current methods impressively generate realistic dialogue in isolated modalities like speech or vision, challenges remain in controllable Multimodal Dialogue Generation (MDG). This paper focuses on the natural alignment between speech, vision, and text in human interaction, aiming for expressive dialogue generation through multimodal conditional control. To address the insufficient richness and diversity of dialogue expressiveness in existing datasets, we introduce a novel multimodal dialogue annotation pipeline to curate dialogues from movies and TV series with fine-grained annotations in interactional characteristics. The resulting MM-Dia dataset (360+ hours, 54,700 dialogues) facilitates explicitly controlled MDG, specifically through style-controllable dialogue speech synthesis. In parallel, MM-Dia-Bench (309 highly expressive dialogues with visible single-/dual-speaker scenes) serves as a rigorous testbed for implicit cross-modal MDG control, evaluating audio-visual style consistency across modalities. Extensive experiments demonstrate that training on MM-Dia significantly enhances fine-grained controllability, while evaluations on MM-Dia-Bench reveal limitations in current frameworks to replicate the nuanced expressiveness of human interaction. These findings provides new insights and challenges for multimodal conditional dialogue generation.

翻译：人工智能生成内容（AIGC）的最新进展在模拟人类交互方面取得了显著进步，尤其是在多模态对话的语境中。尽管当前方法在语音或视觉等孤立模态中能够令人印象深刻地生成逼真的对话，但在可控多模态对话生成（MDG）方面仍存在挑战。本文聚焦于人类交互中语音、视觉与文本之间的自然对齐，旨在通过多模态条件控制实现富有表现力的对话生成。为解决现有数据集中对话表现力丰富性和多样性的不足，我们提出了一种新颖的多模态对话标注流水线，从电影和电视剧中筛选出具有细粒度交互特征标注的对话。由此产生的MM-Dia数据集（360+小时，54,700个对话）促进了显式可控的MDG，特别是通过风格可控的对话语音合成。与此同时，MM-Dia-Bench（309个具有高表现力的对话，包含可见的单/双说话人场景）作为隐式跨模态MDG控制的严格测试平台，评估了跨模态的音视频风格一致性。大量实验表明，在MM-Dia上训练显著增强了细粒度可控性，而对MM-Dia-Bench的评估则揭示了当前框架在复现人类交互细腻表现力方面的局限性。这些发现为多模态条件对话生成提供了新的见解和挑战。