Most current multi-modal summarization methods follow a cascaded manner, where an off-the-shelf object detector is first used to extract visual features, then these features are fused with language representations to generate the summary with an encoder-decoder model. The cascaded way cannot capture the semantic alignments between images and paragraphs, which are crucial to a precise summary. In this paper, we propose ViL-Sum to jointly model paragraph-level \textbf{Vi}sion-\textbf{L}anguage Semantic Alignment and Multi-Modal \textbf{Sum}marization. The core of ViL-Sum is a joint multi-modal encoder with two well-designed tasks, image reordering and image selection. The joint multi-modal encoder captures the interactions between modalities, where the reordering task guides the model to learn paragraph-level semantic alignment and the selection task guides the model to selected summary-related images in the final summary. Experimental results show that our proposed ViL-Sum significantly outperforms current state-of-the-art methods. In further analysis, we find that two well-designed tasks and joint multi-modal encoder can effectively guide the model to learn reasonable paragraphs-images and summary-images relations.

翻译：当前大多数多模态摘要方法采用级联方式，即先使用现成的目标检测器提取视觉特征，再将这些特征与语言表征融合，通过编码器-解码器模型生成摘要。这种级联方式无法捕捉图像与段落之间对精确摘要至关重要的语义对齐关系。本文提出ViL-Sum模型，联合建模段落级视觉-语言语义对齐与多模态摘要。ViL-Sum的核心是一个联合多模态编码器，配备两个精心设计的任务：图像重排序与图像选择。联合多模态编码器捕捉模态间的交互，其中重排序任务引导模型学习段落级语义对齐，选择任务则引导模型为最终摘要选取与摘要相关的图像。实验结果表明，所提出的ViL-Sum显著优于当前最先进方法。进一步分析发现，这两个精心设计的任务与联合多模态编码器可有效引导模型学习合理的段落-图像及摘要-图像关联关系。