We formally define a novel valuable information retrieval task: image-to-multi-modal-retrieval (IMMR), where the query is an image and the doc is an entity with both image and textual description. IMMR task is valuable in various industrial application. We analyze three key challenges for IMMR: 1) skewed data and noisy label in metric learning, 2) multi-modality fusion, 3) effective and efficient training in large-scale industrial scenario. To tackle the above challenges, we propose a novel framework for IMMR task. Our framework consists of three components: 1) a novel data governance scheme coupled with a large-scale classification-based learning paradigm. 2) model architecture specially designed for multimodal learning, where the proposed concept-aware modality fusion module adaptively fuse image and text modality. 3. a hybrid parallel training approach for tackling large-scale training in industrial scenario. The proposed framework achieves SOTA performance on public datasets and has been deployed in a real-world industrial search system, leading to significant improvements in click-through rate and deal number. Code and data will be made publicly available.

翻译：本文正式定义了一项新颖且有价值的信息检索任务：图像到多模态检索（IMMR），其中查询为图像，文档为包含图像和文本描述的实体。IMMR任务在多种工业应用中具有重要价值。我们分析了IMMR面临的三个关键挑战：1）度量学习中的偏斜数据和噪声标签，2）多模态融合，3）大规模工业场景下的高效训练。为应对上述挑战，我们提出了一种面向IMMR任务的新框架。该框架包含三个组成部分：1）一种新颖的数据治理方案，结合大规模基于分类的学习范式；2）专为多模态学习设计的模型架构，其中提出的概念感知模态融合模块可自适应融合图像和文本模态；3）一种混合并行训练方法，用于应对工业场景中的大规模训练。所提框架在公开数据集上取得了最先进的性能，并已部署于真实工业搜索系统中，显著提升了点击率和成交单数。代码与数据将公开发布。