Visual storytelling aims to generate compelling narratives from image sequences. Existing models often focus on enhancing the representation of the image sequence, e.g., with external knowledge sources or advanced graph structures. Despite recent progress, the stories are often repetitive, illogical, and lacking in detail. To mitigate these issues, we present a novel framework which integrates visual representations with pretrained language models and planning. Our model translates the image sequence into a visual prefix, a sequence of continuous embeddings which language models can interpret. It also leverages a sequence of question-answer pairs as a blueprint plan for selecting salient visual concepts and determining how they should be assembled into a narrative. Automatic and human evaluation on the VIST benchmark (Huang et al., 2016) demonstrates that blueprint-based models generate stories that are more coherent, interesting, and natural compared to competitive baselines and state-of-the-art systems.

翻译：视觉叙事旨在从图像序列中生成引人入胜的叙述。现有模型通常侧重于增强图像序列的表示，例如借助外部知识源或先进的图结构。尽管近期取得了进展，但生成的叙事往往存在重复、不合逻辑且缺乏细节的问题。为缓解这些问题，我们提出了一种新颖的框架，该框架将视觉表示与预训练语言模型及规划相结合。我们的模型将图像序列转换为视觉前缀，即语言模型可解释的连续嵌入序列。同时，它利用问答对序列作为蓝图计划，用于选择显著的视觉概念并确定如何将其组织成叙事。在VIST基准测试（Huang等人，2016）上的自动评估和人工评估表明，与具有竞争力的基线及最先进的系统相比，基于蓝图的模型生成的叙事更加连贯、有趣且自然。