Most current multi-modal summarization methods follow a cascaded manner, where an off-the-shelf object detector is first used to extract visual features, then these features are fused with language representations to generate the summary with an encoder-decoder model. The cascaded way cannot capture the semantic alignments between images and paragraphs, which are crucial to a precise summary. In this paper, we propose ViL-Sum to jointly model paragraph-level \textbf{Vi}sion-\textbf{L}anguage Semantic Alignment and Multi-Modal \textbf{Sum}marization. The core of ViL-Sum is a joint multi-modal encoder with two well-designed tasks, image reordering and image selection. The joint multi-modal encoder captures the interactions between modalities, where the reordering task guides the model to learn paragraph-level semantic alignment and the selection task guides the model to selected summary-related images in the final summary. Experimental results show that our proposed ViL-Sum significantly outperforms current state-of-the-art methods. In further analysis, we find that two well-designed tasks and joint multi-modal encoder can effectively guide the model to learn reasonable paragraphs-images and summary-images relations.

翻译：当前多数多模态摘要方法采用级联方式，即先使用现成物体检测器提取视觉特征，再将这些特征与语言表示融合，通过编码器-解码器模型生成摘要。这种级联方式无法捕捉图像与段落间的语义对齐关系，而这对生成精确摘要至关重要。本文提出ViL-Sum模型，联合建模段落级视觉-语言语义对齐与多模态摘要。ViL-Sum的核心是联合多模态编码器及两个精心设计的任务：图像重排序与图像选择。联合多模态编码器捕获模态间的交互，其中重排序任务引导模型学习段落级语义对齐，选择任务引导模型在最终摘要中选取与摘要相关的图像。实验结果表明，本文提出的ViL-Sum显著优于当前最先进方法。进一步分析发现，两个精心设计的任务与联合多模态编码器能有效引导模型学习合理的段落-图像与摘要-图像关系。