Goal-oriented generative script learning aims to generate subsequent steps based on a goal, which is an essential task to assist robots in performing stereotypical activities of daily life. We show that the performance of this task can be improved if historical states are not just captured by the linguistic instructions given to people, but are augmented with the additional information provided by accompanying images. Therefore, we propose a new task, Multimedia Generative Script Learning, to generate subsequent steps by tracking historical states in both text and vision modalities, as well as presenting the first benchmark containing 2,338 tasks and 31,496 steps with descriptive images. We aim to generate scripts that are visual-state trackable, inductive for unseen tasks, and diverse in their individual steps. We propose to encode visual state changes through a multimedia selective encoder, transferring knowledge from previously observed tasks using a retrieval-augmented decoder, and presenting the distinct information at each step by optimizing a diversity-oriented contrastive learning objective. We define metrics to evaluate both generation quality and inductive quality. Experiment results demonstrate that our approach significantly outperforms strong baselines.

翻译：面向目标生成脚本学习旨在根据目标生成后续步骤，这是协助机器人完成日常生活典型活动的关键任务。我们表明，若历史状态不仅通过面向人类的语言指令来捕捉，还通过伴随图像提供的额外信息进行增强，该任务的性能可得到提升。因此，我们提出新任务"多媒体生成式脚本学习"，通过追踪文本与视觉两种模态的历史状态来生成后续步骤，并首次构建包含2,338个任务及31,496个带描述性图像步骤的基准数据集。我们的目标在于生成具备视觉状态可追踪性、面向未见任务的归纳性以及步骤多样性的脚本。我们提出通过多媒体选择性编码器编码视觉状态变化，利用检索增强解码器迁移先前观测任务的知识，并通过优化面向多样性的对比学习目标呈现各步骤的独特信息。我们定义了评估生成质量与归纳质量的指标，实验结果表明我们的方法显著优于强基线方法。