Deep neural networks have achieved promising results in automatic image captioning due to their effective representation learning and context-based content generation capabilities. As a prominent type of deep features used in many of the recent image captioning methods, the well-known bottomup features provide a detailed representation of different objects of the image in comparison with the feature maps directly extracted from the raw image. However, the lack of high-level semantic information about the relationships between these objects is an important drawback of bottom-up features, despite their expensive and resource-demanding extraction procedure. To take advantage of visual relationships in caption generation, this paper proposes a deep neural network architecture for image captioning based on fusing the visual relationships information extracted from an image's scene graph with the spatial feature maps of the image. A multi-modal reward function is then introduced for deep reinforcement learning of the proposed network using a combination of language and vision similarities in a common embedding space. The results of extensive experimentation on the MSCOCO dataset show the effectiveness of using visual relationships in the proposed captioning method. Moreover, the results clearly indicate that the proposed multi-modal reward in deep reinforcement learning leads to better model optimization, outperforming several state-of-the-art image captioning algorithms, while using light and easy to extract image features. A detailed experimental study of the components constituting the proposed method is also presented.

翻译：深度神经网络凭借其有效的表征学习与基于上下文的生成能力，在自动图像描述任务中取得了令人瞩目的成果。作为近期许多图像描述方法中广泛使用的一种典型深度特征，著名的自底向上特征相较于直接从原始图像提取的特征图，能够提供图像中不同对象的详细表征。然而，尽管自底向上特征的提取过程昂贵且资源密集，其缺乏关于对象间关系的高层语义信息却是一个重要缺陷。为在描述生成中利用视觉关系，本文提出一种融合图像场景图提取的视觉关系信息与空间特征图的深度神经网络架构用于图像描述生成。进而，通过将语言相似性与视觉相似性在共同嵌入空间中进行融合，为所提网络的深度强化学习引入了多模态奖励函数。在MSCOCO数据集上的大量实验结果表明，在所提描述方法中利用视觉关系具有有效性。此外，结果清晰表明，所提出的深度强化学习中的多模态奖励能带来更优的模型优化，在采用轻量且易于提取的图像特征的同时，该方法超越了多项最先进的图像描述算法。本文还对构成所提方法的各组件进行了详细的实验研究。