Image-guided story ending generation (IgSEG) is to generate a story ending based on given story plots and ending image. Existing methods focus on cross-modal feature fusion but overlook reasoning and mining implicit information from story plots and ending image. To tackle this drawback, we propose a multimodal event transformer, an event-based reasoning framework for IgSEG. Specifically, we construct visual and semantic event graphs from story plots and ending image, and leverage event-based reasoning to reason and mine implicit information in a single modality. Next, we connect visual and semantic event graphs and utilize cross-modal fusion to integrate different-modality features. In addition, we propose a multimodal injector to adaptive pass essential information to decoder. Besides, we present an incoherence detection to enhance the understanding context of a story plot and the robustness of graph modeling for our model. Experimental results show that our method achieves state-of-the-art performance for the image-guided story ending generation.

翻译：图像引导的故事结尾生成（IgSEG）的目标是根据给定的故事情节和结尾图像生成故事结局。现有方法侧重于跨模态特征融合，但忽略了从故事情节和结尾图像中推理和挖掘隐含信息。为解决这一缺陷，我们提出了一种多模态事件Transformer——一种基于事件推理的IgSEG框架。具体而言，我们从故事情节和结尾图像中构建视觉和语义事件图，并利用基于事件的推理在单一模态中推理和挖掘隐含信息。接着，我们连接视觉和语义事件图，并通过跨模态融合整合不同模态的特征。此外，我们提出了一种多模态注入器，以自适应地将关键信息传递给解码器。另外，我们引入了一种不连贯检测机制，以增强模型对故事情节上下文的理解以及图建模的鲁棒性。实验结果表明，我们的方法在图像引导的故事结尾生成任务上取得了最先进的性能。