Video-to-speech synthesis is the task of reconstructing the speech signal from a silent video of a speaker. Most established approaches to date involve a two-step process, whereby an intermediate representation from the video, such as a spectrogram, is extracted first and then passed to a vocoder to produce the raw audio. Some recent work has focused on end-to-end synthesis, whereby the generation of raw audio and any intermediate representations is performed jointly. All such approaches involve training on data from almost exclusively audio-visual datasets, i.e. every audio sample has a corresponding video sample. This precludes the use of abundant audio-only datasets which may not have a corresponding visual modality (e.g. audiobooks, radio podcasts, speech recognition datasets etc.), as well as audio-only architectures that have been developed by the audio machine learning community over the years. In this paper we propose to train encoder-decoder models on more than 3,500 hours of audio data at 24kHz, and then use the pre-trained decoders to initialize the audio decoders for the video-to-speech synthesis task. The pre-training step uses audio samples only and does not require labels or corresponding samples from other modalities (visual, text). We demonstrate that this pre-training step improves the reconstructed speech and that it is an unexplored way to improve the quality of the generator in a cross-modal task while only requiring samples from one of the modalities. We conduct experiments using both raw audio and mel spectrograms as target outputs and benchmark our models with existing work.

翻译：视频到语音合成是从说话者的无声视频中重建语音信号的任务。迄今为止，大多数已建立的方法涉及两步流程，首先从视频中提取中间表征（如频谱图），然后将其传递给声码器以生成原始音频。近期一些工作聚焦于端到端合成，即联合执行原始音频生成与任何中间表征的生成。所有这些方法几乎都完全依赖音视频数据集进行训练，即每个音频样本都有对应的视频样本。这排除了可能没有相应视觉模态的海量纯音频数据集（如有声书、广播播客、语音识别数据集等），以及音频机器学习社区多年来开发的纯音频架构。本文提出在超过3,500小时、采样率为24kHz的音频数据上训练编码器-解码器模型，而后利用预训练的解码器初始化视频到语音合成任务的音频解码器。预训练步骤仅使用音频样本，无需标签或其他模态（视觉、文本）的对应样本。我们证明该预训练步骤能提升重建语音质量，并揭示这是一种通过仅需单一模态样本来改善跨模态任务生成器质量的未探索途径。我们使用原始音频和梅尔频谱图作为目标输出进行实验，并将我们的模型与现有工作进行基准对比。