Knowledge retrieval with multi-modal queries plays a crucial role in supporting knowledge-intensive multi-modal applications. However, existing methods face challenges in terms of their effectiveness and training efficiency, especially when it comes to training and integrating multiple retrievers to handle multi-modal queries. In this paper, we propose an innovative end-to-end generative framework for multi-modal knowledge retrieval. Our framework takes advantage of the fact that large language models (LLMs) can effectively serve as virtual knowledge bases, even when trained with limited data. We retrieve knowledge via a two-step process: 1) generating knowledge clues related to the queries, and 2) obtaining the relevant document by searching databases using the knowledge clue. In particular, we first introduce an object-aware prefix-tuning technique to guide multi-grained visual learning. Then, we align multi-grained visual features into the textual feature space of the LLM, employing the LLM to capture cross-modal interactions. Subsequently, we construct instruction data with a unified format for model training. Finally, we propose the knowledge-guided generation strategy to impose prior constraints in the decoding steps, thereby promoting the generation of distinctive knowledge clues. Through experiments conducted on three benchmarks, we demonstrate significant improvements ranging from 3.0% to 14.6% across all evaluation metrics when compared to strong baselines.

翻译：多模态查询的知识检索在支持知识密集型多模态应用中发挥着关键作用。然而，现有方法在有效性及训练效率方面面临挑战，尤其是在训练和集成多个检索器以处理多模态查询时。本文提出一种创新的端到端生成式多模态知识检索框架。该框架利用大语言模型（LLMs）即使在有限数据训练下也能有效充当虚拟知识库的特性。我们通过两步流程实现知识检索：1）生成与查询相关的知识线索；2）利用该线索通过数据库搜索获取相关文档。具体而言，我们首先引入一种对象感知的前缀调优技术，以引导多粒度视觉学习。随后，将多粒度视觉特征对齐至LLM的文本特征空间，利用LLM捕捉跨模态交互。接着，构造统一格式的指令数据进行模型训练。最后，提出知识引导生成策略，在解码步骤中施加先验约束，从而促进区分性知识线索的生成。通过在三个基准数据集上的实验，我们证明与强基线相比，所有评估指标均实现了3.0%至14.6%的显著提升。