This paper proposes a novel direct Audio-Visual Speech to Audio-Visual Speech Translation (AV2AV) framework, where the input and output of the system are multimodal (i.e., audio and visual speech). With the proposed AV2AV, two key advantages can be brought: 1) We can perform real-like conversations with individuals worldwide in a virtual meeting by utilizing our own primary languages. In contrast to Speech-to-Speech Translation (A2A), which solely translates between audio modalities, the proposed AV2AV directly translates between audio-visual speech. This capability enhances the dialogue experience by presenting synchronized lip movements along with the translated speech. 2) We can improve the robustness of the spoken language translation system. By employing the complementary information of audio-visual speech, the system can effectively translate spoken language even in the presence of acoustic noise, showcasing robust performance. To mitigate the problem of the absence of a parallel AV2AV translation dataset, we propose to train our spoken language translation system with the audio-only dataset of A2A. This is done by learning unified audio-visual speech representations through self-supervised learning in advance to train the translation system. Moreover, we propose an AV-Renderer that can generate raw audio and video in parallel. It is designed with zero-shot speaker modeling, thus the speaker in source audio-visual speech can be maintained at the target translated audio-visual speech. The effectiveness of AV2AV is evaluated with extensive experiments in a many-to-many language translation setting. The demo page is available on https://choijeongsoo.github.io/av2av.

翻译：本文提出了一种新颖的直接音视频语音到音视频语音翻译（AV2AV）框架，其系统输入和输出均为多模态形式（即音频和视觉语音）。所提出的AV2AV框架可带来两大关键优势：1）在虚拟会议中，我们可通过使用各自的母语，与全球各地人士进行类真实对话。与仅进行音频模态间翻译的语音到语音翻译（A2A）不同，本文提出的AV2AV直接在音视频语音间进行翻译。该能力通过呈现与翻译语音同步的唇部运动，显著提升了对话体验。2）可增强口语翻译系统的鲁棒性。通过利用音视频语音的互补信息，系统即便在存在声学噪声的环境下也能有效完成口语翻译，展现出鲁棒性能。为解决并行AV2AV翻译数据集缺失问题，我们提出利用A2A的纯音频数据集训练口语翻译系统。其实现路径是：预先通过自监督学习获得统一的音视频语音表征，进而训练翻译系统。此外，我们提出可并行生成原始音频与视频的AV渲染器，该组件采用零样本说话人建模设计，使得源音视频语音中的说话人特征得以保留至目标翻译音视频中。我们通过在多对多语言翻译场景下的大量实验验证了AV2AV的有效性。演示页面请访问https://choijeongsoo.github.io/av2av。