Recently video generation has achieved substantial progress with realistic results. Nevertheless, existing AI-generated videos are usually very short clips ("shot-level") depicting a single scene. To deliver a coherent long video ("story-level"), it is desirable to have creative transition and prediction effects across different clips. This paper presents a short-to-long video diffusion model, SEINE, that focuses on generative transition and prediction. The goal is to generate high-quality long videos with smooth and creative transitions between scenes and varying lengths of shot-level videos. Specifically, we propose a random-mask video diffusion model to automatically generate transitions based on textual descriptions. By providing the images of different scenes as inputs, combined with text-based control, our model generates transition videos that ensure coherence and visual quality. Furthermore, the model can be readily extended to various tasks such as image-to-video animation and autoregressive video prediction. To conduct a comprehensive evaluation of this new generative task, we propose three assessing criteria for smooth and creative transition: temporal consistency, semantic similarity, and video-text semantic alignment. Extensive experiments validate the effectiveness of our approach over existing methods for generative transition and prediction, enabling the creation of story-level long videos. Project page: https://vchitect.github.io/SEINE-project/ .

翻译：近期视频生成在逼真性方面取得了显著进展，但现有AI生成的视频通常为描绘单一场景的短片段（"镜头级"）。为了实现连贯的长视频（"故事级"），需要在不同片段间实现具有创造性的过渡与预测效果。本文提出一种短到长视频扩散模型SEINE，专注于生成式过渡与预测。其目标是在场间及可变长度镜头级视频间生成具有平滑且创造性过渡的高质量长视频。具体而言，我们提出一种随机掩码视频扩散模型，可基于文本描述自动生成过渡片段。通过将不同场景图像作为输入，结合文本控制，该模型可生成确保连贯性与视觉质量的过渡视频。此外，该模型可便捷扩展至图像到视频动画、自回归视频预测等多种任务。为系统评估这一新型生成任务，我们提出三项评估标准用于衡量平滑且创造性的过渡：时间一致性、语义相似性及视频-文本语义对齐。大量实验验证了本方法在生成式过渡与预测任务中相较现有方法的有效性，从而支持了故事级长视频的创作。项目页面：https://vchitect.github.io/SEINE-project/。