Analogical reasoning is fundamental to human cognition and holds an important place in various fields. However, previous studies mainly focus on single-modal analogical reasoning and ignore taking advantage of structure knowledge. Notably, the research in cognitive psychology has demonstrated that information from multimodal sources always brings more powerful cognitive transfer than single modality sources. To this end, we introduce the new task of multimodal analogical reasoning over knowledge graphs, which requires multimodal reasoning ability with the help of background knowledge. Specifically, we construct a Multimodal Analogical Reasoning dataSet (MARS) and a multimodal knowledge graph MarKG. We evaluate with multimodal knowledge graph embedding and pre-trained Transformer baselines, illustrating the potential challenges of the proposed task. We further propose a novel model-agnostic Multimodal analogical reasoning framework with Transformer (MarT) motivated by the structure mapping theory, which can obtain better performance. Code and datasets are available in https://github.com/zjunlp/MKG_Analogy.

翻译：类比推理是人类认知的基础，在多个领域具有重要地位。然而，先前研究主要集中于单模态类比推理，且忽视了结构知识的利用。值得注意的是，认知心理学研究表明，多模态信息源相比单模态信息源总能带来更强大的认知迁移。为此，我们提出了知识图谱上的多模态类比推理新任务，该任务需要借助背景知识实现多模态推理能力。具体而言，我们构建了多模态类比推理数据集（MARS）与多模态知识图谱MarKG，通过多模态知识图谱嵌入和预训练Transformer基线模型评估了该任务的潜在挑战。受结构映射理论启发，我们进一步提出了一种与模型无关的多模态类比推理框架（MarT），该框架可取得更优性能。代码与数据集已开源至https://github.com/zjunlp/MKG_Analogy。