Despite recent advances of AI, story understanding remains an open and under-investigated problem. We collect, preprocess, and publicly release a video-language story dataset, Synopses of Movie Narratives (SyMoN), containing 5,193 video summaries of popular movies and TV series with a total length of 869 hours. SyMoN captures naturalistic storytelling videos made by human creators and intended for a human audience. As a prototypical and naturalistic story dataset, SyMoN features high coverage of multimodal story events and abundant mental-state descriptions. Its use of storytelling techniques cause cross-domain semantic gaps that provide appropriate challenges to existing models. We establish benchmarks on video-text retrieval and zero-shot alignment on movie summary videos, which showcase the importance of in-domain data and long-term memory in story understanding. With SyMoN, we hope to lay the groundwork for progress in multimodal story understanding.

翻译：尽管人工智能近期取得进展，故事理解仍是一个未充分探索的开放性问题。我们收集、预处理并公开发布了一个视频语言故事数据集——电影叙事概要（SyMoN），包含5,193个流行电影和电视剧的视频摘要，总时长达869小时。SyMoN捕捉了由人类创作者制作、面向人类受众的自然叙事视频。作为典型且自然的故事情数据集，SyMoN具有多模态故事事件的高覆盖率和丰富的心理状态描述特点。其采用的叙事技巧导致跨域语义差距，为现有模型提供了恰当的挑战。我们在电影摘要视频上建立了视频-文本检索和零样本对齐基准测试，这些测试展示了领域内数据和长时记忆在故事理解中的重要性。借助SyMoN，我们期望为多模态故事理解的进步奠定基础。