Multi-modal relation extraction (MMRE) is a challenging task that aims to identify relations between entities in text leveraging image information. Existing methods are limited by their neglect of the multiple entity pairs in one sentence sharing very similar contextual information (ie, the same text and image), resulting in increased difficulty in the MMRE task. To address this limitation, we propose the Variational Multi-Modal Hypergraph Attention Network (VM-HAN) for multi-modal relation extraction. Specifically, we first construct a multi-modal hypergraph for each sentence with the corresponding image, to establish different high-order intra-/inter-modal correlations for different entity pairs in each sentence. We further design the Variational Hypergraph Attention Networks (V-HAN) to obtain representational diversity among different entity pairs using Gaussian distribution and learn a better hypergraph structure via variational attention. VM-HAN achieves state-of-the-art performance on the multi-modal relation extraction task, outperforming existing methods in terms of accuracy and efficiency.

翻译：多模态关系抽取（MMRE）是一项具有挑战性的任务，旨在利用图像信息识别文本中实体之间的关系。现有方法受限于其忽略了同一句子中多个实体对共享非常相似的上下文信息（即相同的文本和图像），从而增加了MMRE任务的难度。为解决这一局限性，我们提出了变分多模态超图注意力网络（VM-HAN）用于多模态关系抽取。具体而言，我们首先为每个句子及其对应图像构建一个多模态超图，以建立句子中不同实体对之间的高阶模态内/跨模态关联。进一步设计了变分超图注意力网络（V-HAN），通过高斯分布获得不同实体对之间的表示多样性，并通过变分注意力学习更优的超图结构。VM-HAN在多模态关系抽取任务上达到了最先进的性能，在准确性和效率方面均优于现有方法。