Lip-to-speech involves generating a natural-sounding speech synchronized with a soundless video of a person talking. Despite recent advances, current methods still cannot produce high-quality speech with high levels of intelligibility for challenging and realistic datasets such as LRS3. In this work, we present LipVoicer, a novel method that generates high-quality speech, even for in-the-wild and rich datasets, by incorporating the text modality. Given a silent video, we first predict the spoken text using a pre-trained lip-reading network. We then condition a diffusion model on the video and use the extracted text through a classifier-guidance mechanism where a pre-trained ASR serves as the classifier. LipVoicer outperforms multiple lip-to-speech baselines on LRS2 and LRS3, which are in-the-wild datasets with hundreds of unique speakers in their test set and an unrestricted vocabulary. Moreover, our experiments show that the inclusion of the text modality plays a major role in the intelligibility of the produced speech, readily perceptible while listening, and is empirically reflected in the substantial reduction of the WER metric. We demonstrate the effectiveness of LipVoicer through human evaluation, which shows that it produces more natural and synchronized speech signals compared to competing methods. Finally, we created a demo showcasing LipVoicer's superiority in producing natural, synchronized, and intelligible speech, providing additional evidence of its effectiveness. Project page: https://lipvoicer.github.io

翻译：唇语转语音涉及从无声的人物说话视频中生成自然同步的语音信号。尽管近年取得进展，现有方法仍无法针对LRS3等具有挑战性的真实数据集生成高清晰度与高可懂度的语音。本文提出LipVoicer，一种通过融合文本模态生成高质量语音的新方法，即使在野外复杂数据集中亦表现优异。给定无声视频，我们首先利用预训练唇读网络预测口语文本，然后以视频为条件构建扩散模型，并通过预训练ASR作为分类器的分类器引导机制利用提取的文本。LipVoicer在LRS2和LRS3数据集上超越多项唇语转语音基线方法，这些数据集包含数百个独特说话者及无限制词汇的野外测试集。此外，实验表明文本模态的引入对生成语音的可懂度起关键作用，该提升在听觉上即时可辨，且实证体现在WER指标的显著降低。通过人工评估，我们证实LipVoicer相比竞争方法能生成更自然且同步的语音信号。最后，我们创建了演示示例展示LipVoicer在生成自然、同步且清晰语音方面的优越性，为其有效性提供额外证据。项目页面：https://lipvoicer.github.io