Multi-modal retrieval has seen tremendous progress with the development of vision-language models. However, further improving these models require additional labelled data which is a huge manual effort. In this paper, we propose a framework MuMUR, that utilizes knowledge transfer from a multilingual model to boost the performance of multi-modal (image and video) retrieval. We first use state-of-the-art machine translation models to construct pseudo ground-truth multilingual visual-text pairs. We then use this data to learn a joint vision-text representation where English and non-English text queries are represented in a common embedding space based on pretrained multilingual models. We evaluate our proposed approach on a diverse set of retrieval datasets: five video retrieval datasets such as MSRVTT, MSVD, DiDeMo, Charades and MSRVTT multilingual, two image retrieval datasets such as Flickr30k and Multi30k . Experimental results demonstrate that our approach achieves state-of-the-art results on all video retrieval datasets outperforming previous models. Additionally, our framework MuMUR significantly beats other multilingual video retrieval dataset. We also observe that MuMUR exhibits strong performance on image retrieval. This demonstrates the universal ability of MuMUR to perform retrieval across all visual inputs (image and video) and text inputs (monolingual and multilingual).

翻译：多模态检索随着视觉语言模型的发展取得了巨大进展。然而，进一步提升这些模型需要额外标注数据，而这需要大量人工投入。本文提出了一种名为MuMUR的框架，它利用多语言模型的知识迁移来提升多模态（图像和视频）检索的性能。我们首先使用最先进的机器翻译模型构建伪真实多语言视觉-文本对，然后利用这些数据学习联合视觉-文本表示，使英语和非英语文本查询基于预训练的多语言模型共享同一嵌入空间。我们在多种检索数据集上评估了所提方法：包括MSRVTT、MSVD、DiDeMo、Charades和MSRVTT多语言在内的五个视频检索数据集，以及Flickr30k和Multi30k两个图像检索数据集。实验结果表明，我们的方法在所有视频检索数据集上均取得了最先进的结果，超越了先前模型。此外，MuMUR框架显著优于其他多语言视频检索数据集。我们还观察到MuMUR在图像检索上表现出强劲性能。这证明了MuMUR在跨所有视觉输入（图像和视频）与文本输入（单语言和多语言）进行检索时的通用能力。