Cross-lingual self-supervised learning has been a growing research topic in the last few years. However, current works only explored the use of audio signals to create representations. In this work, we study cross-lingual self-supervised visual representation learning. We use the recently-proposed Raw Audio-Visual Speech Encoders (RAVEn) framework to pre-train an audio-visual model with unlabelled multilingual data, and then fine-tune the visual model on labelled transcriptions. Our experiments show that: (1) multi-lingual models with more data outperform monolingual ones, but, when keeping the amount of data fixed, monolingual models tend to reach better performance; (2) multi-lingual outperforms English-only pre-training; (3) using languages which are more similar yields better results; and (4) fine-tuning on unseen languages is competitive to using the target language in the pre-training set. We hope our study inspires future research on non-English-only speech representation learning.

翻译：跨语言自监督学习近年来已成为一个日益增长的研究课题。然而，当前的研究仅探索了利用音频信号来创建表征。在本工作中，我们研究了跨语言自监督视觉表征学习。我们使用近期提出的原始音频-视觉语音编码器（RAVEn）框架，利用未标注的多语言数据预训练一个音频-视觉模型，然后在标注的转录文本上对视觉模型进行微调。我们的实验结果表明：（1）使用更多数据的多语言模型优于单语言模型，但当数据量固定时，单语言模型往往能达到更优性能；（2）多语言预训练优于仅使用英语的预训练；（3）使用语言相似度更高的语言能获得更好的结果；（4）在未见过的语言上进行微调的效果与在预训练集中包含目标语言的效果相当。我们希望本研究能启发未来对非英语中心语音表征学习的进一步探索。