Multimodal named entity recognition (MNER) and multimodal relation extraction (MRE) are two fundamental subtasks in the multimodal knowledge graph construction task. However, the existing methods usually handle two tasks independently, which ignores the bidirectional interaction between them. This paper is the first to propose jointly performing MNER and MRE as a joint multimodal entity-relation extraction task (JMERE). Besides, the current MNER and MRE models only consider aligning the visual objects with textual entities in visual and textual graphs but ignore the entity-entity relationships and object-object relationships. To address the above challenges, we propose an edge-enhanced graph alignment network and a word-pair relation tagging (EEGA) for JMERE task. Specifically, we first design a word-pair relation tagging to exploit the bidirectional interaction between MNER and MRE and avoid the error propagation. Then, we propose an edge-enhanced graph alignment network to enhance the JMERE task by aligning nodes and edges in the cross-graph. Compared with previous methods, the proposed method can leverage the edge information to auxiliary alignment between objects and entities and find the correlations between entity-entity relationships and object-object relationships. Experiments are conducted to show the effectiveness of our model.

翻译：多模态命名实体识别（MNER）与多模态关系抽取（MRE）是多模态知识图谱构建任务中的两个基础子任务。然而，现有方法通常独立处理这两个任务，忽视了它们之间的双向交互。本文首次提出将MNER与MRE联合执行，作为联合多模态实体关系抽取任务（JMERE）。此外，当前MNER和MRE模型仅考虑在视觉和文本图中将视觉对象与文本实体对齐，却忽略了实体-实体关系与对象-对象关系。针对上述挑战，我们提出了一种边缘增强图对齐网络与词对关系标注（EEGA）方法用于JMERE任务。具体地，我们首先设计词对关系标注以利用MNER与MRE之间的双向交互，并避免误差传播。随后提出边缘增强图对齐网络，通过跨图节点与边缘对齐来增强JMERE任务。与先前方法相比，本方法可利用边缘信息辅助对象与实体间的对齐，并发现实体-实体关系与对象-对象关系之间的关联性。实验结果表明了该模型的有效性。