Multimedia generation approaches occupy a prominent place in artificial intelligence research. Text-to-image models achieved high-quality results over the last few years. However, video synthesis methods recently started to develop. This paper presents a new two-stage latent diffusion text-to-video generation architecture based on the text-to-image diffusion model. The first stage concerns keyframes synthesis to figure the storyline of a video, while the second one is devoted to interpolation frames generation to make movements of the scene and objects smooth. We compare several temporal conditioning approaches for keyframes generation. The results show the advantage of using separate temporal blocks over temporal layers in terms of metrics reflecting video generation quality aspects and human preference. The design of our interpolation model significantly reduces computational costs compared to other masked frame interpolation approaches. Furthermore, we evaluate different configurations of MoVQ-based video decoding scheme to improve consistency and achieve higher PSNR, SSIM, MSE, and LPIPS scores. Finally, we compare our pipeline with existing solutions and achieve top-2 scores overall and top-1 among open-source solutions: CLIPSIM = 0.2976 and FVD = 433.054. Project page: https://ai-forever.github.io/kandinsky-video/

翻译：多媒体生成方法在人工智能研究中占据重要地位。近年来，文本到图像模型已取得高质量成果，但视频合成方法近期才开始发展。本文提出一种基于文本到图像扩散模型的新型两阶段潜扩散文本到视频生成架构。第一阶段涉及关键帧合成以勾勒视频故事情节，第二阶段则专注于插值帧生成，使场景与物体运动更加平滑。我们比较了多种关键帧生成的时序条件方法。结果表明，在反映视频生成质量指标及人类偏好的评估中，使用独立时序模块相比时序层更具优势。与其他掩膜帧插值方法相比，我们的插值模型设计显著降低了计算成本。此外，我们评估了基于MoVQ的视频解码方案的不同配置，以提升一致性并获得更高的PSNR、SSIM、MSE及LPIPS分数。最后，我们将本流水线与现有方案进行比较，整体排名前二，且开源方案中排名第一：CLIPSIM = 0.2976，FVD = 433.054。项目页面：https://ai-forever.github.io/kandinsky-video/