The objective of this paper is to jointly synthesize interactive videos and conversational speech from text and reference images. With the ultimate goal of building human-like conversational systems, recent studies have explored talking or listening head generation as well as conversational speech generation. However, these works are typically studied in isolation, overlooking the multimodal nature of human conversation, which involves tightly coupled audio-visual interactions. In this paper, we introduce TAVID, a unified framework that generates both interactive faces and conversational speech in a synchronized manner. TAVID integrates face and speech generation pipelines through two cross-modal mappers (i.e., a motion mapper and a speaker mapper), which enable bidirectional exchange of complementary information between the audio and visual modalities. We evaluate our system across four dimensions: talking face realism, listening head responsiveness, dyadic interaction fluency, and speech quality. Extensive experiments demonstrate the effectiveness of our approach across all these aspects.

翻译：本文的目标是从文本和参考图像中联合合成交互式视频与会话语音。以构建类人对话系统为最终目标，近期研究已探索了说话或倾听头部生成以及会话语音生成。然而，这些工作通常被孤立研究，忽视了人类对话的多模态本质——其中涉及紧密耦合的视听交互。本文提出TAVID，一个以同步方式生成交互式面部和会话语音的统一框架。TAVID通过两个跨模态映射器（即运动映射器和说话人映射器）整合面部与语音生成流程，这两个映射器实现了音频与视觉模态间互补信息的双向交换。我们从四个维度评估系统性能：说话面部真实感、倾听头部响应性、二元交互流畅度以及语音质量。大量实验证明了我们的方法在所有上述方面的有效性。