Large language models (LLMs) are trained on text-only data that go far beyond the languages with paired speech and text data. At the same time, Dual Encoder (DE) based retrieval systems project queries and documents into the same embedding space and have demonstrated their success in retrieval and bi-text mining. To match speech and text in many languages, we propose using LLMs to initialize multi-modal DE retrieval systems. Unlike traditional methods, our system doesn't require speech data during LLM pre-training and can exploit LLM's multilingual text understanding capabilities to match speech and text in languages unseen during retrieval training. Our multi-modal LLM-based retrieval system is capable of matching speech and text in 102 languages despite only training on 21 languages. Our system outperforms previous systems trained explicitly on all 102 languages. We achieve a 10% absolute improvement in Recall@1 averaged across these languages. Additionally, our model demonstrates cross-lingual speech and text matching, which is further enhanced by readily available machine translation data.

翻译：大型语言模型（LLMs）在纯文本数据上训练，这些数据远超配有语音和文本数据的语言种类。同时，基于双编码器（DE）的检索系统将查询和文档映射到同一嵌入空间，并在检索和双语文本挖掘中展现了其成功之处。为了在多种语言中匹配语音和文本，我们提出使用LLMs来初始化多模态DE检索系统。与传统方法不同，我们的系统在LLM预训练期间不需要语音数据，并且可以利用LLM的多语言文本理解能力，在检索训练中未见过的语言中匹配语音和文本。尽管仅对21种语言进行训练，但基于LLM的多模态检索系统能够匹配102种语言的语音和文本。我们的系统优于先前在所有102种语言上显式训练的系统。在这些语言上，平均Recall@1绝对提高了10%。此外，我们的模型展示了跨语言的语音和文本匹配能力，并通过现有的机器翻译数据进一步增强了这一能力。