This paper proposes a novel lip reading framework, especially for low-resource languages, which has not been well addressed in the previous literature. Since low-resource languages do not have enough video-text paired data to train the model to have sufficient power to model lip movements and language, it is regarded as challenging to develop lip reading models for low-resource languages. In order to mitigate the challenge, we try to learn general speech knowledge, the ability to model lip movements, from a high-resource language through the prediction of speech units. It is known that different languages partially share common phonemes, thus general speech knowledge learned from one language can be extended to other languages. Then, we try to learn language-specific knowledge, the ability to model language, by proposing Language-specific Memory-augmented Decoder (LMDecoder). LMDecoder saves language-specific audio features into memory banks and can be trained on audio-text paired data which is more easily accessible than video-text paired data. Therefore, with LMDecoder, we can transform the input speech units into language-specific audio features and translate them into texts by utilizing the learned rich language knowledge. Finally, by combining general speech knowledge and language-specific knowledge, we can efficiently develop lip reading models even for low-resource languages. Through extensive experiments using five languages, English, Spanish, French, Italian, and Portuguese, the effectiveness of the proposed method is evaluated.

翻译：本文提出一种新颖的唇语识别框架，特别针对以往文献中尚未妥善解决的资源匮乏语言问题。由于低资源语言缺乏足够的视频-文本配对数据来训练模型以充分建模唇部运动与语言特性，开发适用于低资源语言的唇语识别模型被视为一项挑战。为解决这一难题，我们尝试从高资源语言中通过语音单元预测来学习通用语音知识——即建模唇部运动的能力。已知不同语言部分共享共同音素，因此从一种语言习得的通用语音知识可迁移至其他语言。此外，我们提出语言特定记忆增强解码器（LMDecoder），用以学习语言特定知识——即建模语言的能力。LMDecoder可将语言特定音频特征存入记忆库，并利用比视频-文本数据更易获取的音频-文本配对数据进行训练。因此借助LMDecoder，我们可将输入语音单元转化为语言特定音频特征，并通过已习得的丰富语言知识将其转译为文本。最终通过融合通用语音知识与语言特定知识，我们能够高效开发唇语识别模型，即使针对低资源语言亦不例外。通过英语、西班牙语、法语、意大利语和葡萄牙语五种语言的广泛实验，验证了所提方法的有效性。