This paper presents Universal Vision-Language Dense Retrieval (UniVL-DR), which builds a unified model for multi-modal retrieval. UniVL-DR encodes queries and multi-modality resources in an embedding space for searching candidates from different modalities. To learn a unified embedding space for multi-modal retrieval, UniVL-DR proposes two techniques: 1) Universal embedding optimization strategy, which contrastively optimizes the embedding space using the modality-balanced hard negatives; 2) Image verbalization method, which bridges the modality gap between images and texts in the raw data space. UniVL-DR achieves the state-of-the-art on the multi-modal open-domain question answering benchmark, WebQA, and outperforms all retrieval models on the two subtasks, text-text retrieval and text-image retrieval. It demonstrates that universal multi-modal search is feasible to replace the divide-and-conquer pipeline with a united model and also benefits single/cross modality tasks. All source codes of this work are available at https://github.com/OpenMatch/UniVL-DR.

翻译：本文提出通用视觉语言密集检索（UniVL-DR），该模型构建了统一的多模态检索框架。UniVL-DR将查询与多模态资源编码至嵌入空间中，以从不同模态中搜索候选对象。为学习统一的多模态检索嵌入空间，UniVL-DR提出两项关键技术：1）通用嵌入优化策略——利用模态平衡的困难负样本对嵌入空间进行对比优化；2）图像语言化方法——在原始数据空间中弥合图像与文本之间的模态鸿沟。UniVL-DR在多模态开放域问答基准WebQA上取得了最优性能，并在其两个子任务（文本-文本检索与文本-图像检索）中超越所有检索模型。实验表明，通用多模态搜索能够以统一模型替代分治处理流程，同时惠及单模态/跨模态任务。本工作所有源代码已开源至https://github.com/OpenMatch/UniVL-DR。