Cross-lingual image captioning is confronted with both cross-lingual and cross-modal challenges for multimedia analysis. The crucial issue in this task is to model the global and local matching between the image and different languages. Existing cross-modal embedding methods based on Transformer architecture oversight the local matching between the image region and monolingual words, not to mention in the face of a variety of differentiated languages. Due to the heterogeneous property of the cross-modal and cross-lingual task, we utilize the heterogeneous network to establish cross-domain relationships and the local correspondences between the image and different languages. In this paper, we propose an Embedded Heterogeneous Attention Transformer (EHAT) to build reasoning paths bridging cross-domain for cross-lingual image captioning and integrate into transformer. The proposed EHAT consists of a Masked Heterogeneous Cross-attention (MHCA), Heterogeneous Attention Reasoning Network (HARN) and Heterogeneous Co-attention (HCA). HARN as the core network, models and infers cross-domain relationship anchored by vision bounding box representation features to connect two languages word features and learn the heterogeneous maps. MHCA and HCA implement cross-domain integration in the encoder through the special heterogeneous attention and enable single model to generate two language captioning. We test on MSCOCO dataset to generate English and Chinese, which are most widely used and have obvious difference between their language families. Our experiments show that our method even achieve better than advanced monolingual methods.

翻译：跨语言图像描述在多媒体分析中面临跨语言与跨模态的双重挑战。该任务的核心问题在于建模图像与不同语言之间的全局与局部匹配。现有基于Transformer架构的跨模态嵌入方法忽视了图像区域与单语单词之间的局部匹配，更遑论面对多种差异化语言的情况。鉴于跨模态与跨语言任务的异构特性，我们利用异构网络建立跨域关系以及图像与不同语言之间的局部对应关系。本文提出嵌入式异构注意力Transformer（EHAT），为跨语言图像描述构建连接跨域的推理路径，并将其集成到Transformer中。所提出的EHAT由掩码异构交叉注意力（MHCA）、异构注意力推理网络（HARN）和异构共注意力（HCA）组成。HARN作为核心网络，以视觉边界框表征特征为锚点，建模并推理跨域关系，连接两种语言的单词特征并学习异构映射。MHCA和HCA通过特殊的异构注意力在编码器中实现跨域集成，使单一模型能够生成两种语言的图像描述。我们在MSCOCO数据集上进行测试，生成使用最广泛且语系差异显著的英语和中文描述。实验表明，我们的方法甚至优于先进的单语方法。