Multimedia generation approaches occupy a prominent place in artificial intelligence research. Text-to-image models achieved high-quality results over the last few years. However, video synthesis methods recently started to develop. This paper presents a new two-stage latent diffusion text-to-video generation architecture based on the text-to-image diffusion model. The first stage concerns keyframes synthesis to figure the storyline of a video, while the second one is devoted to interpolation frames generation to make movements of the scene and objects smooth. We compare several temporal conditioning approaches for keyframes generation. The results show the advantage of using separate temporal blocks over temporal layers in terms of metrics reflecting video generation quality aspects and human preference. The design of our interpolation model significantly reduces computational costs compared to other masked frame interpolation approaches. Furthermore, we evaluate different configurations of MoVQ-based video decoding scheme to improve consistency and achieve higher PSNR, SSIM, MSE, and LPIPS scores. Finally, we compare our pipeline with existing solutions and achieve top-2 scores overall and top-1 among open-source solutions: CLIPSIM = 0.2976 and FVD = 433.054. Project page: https://ai-forever.github.io/kandinsky-video/

翻译：多媒体生成方法在人工智能研究中占据重要地位。近年来，文本到图像模型已取得高质量成果，但视频合成方法尚处于起步阶段。本文提出一种基于文本到图像扩散模型的新型两阶段隐式扩散文本到视频生成架构。第一阶段涉及关键帧合成以勾勒视频叙事脉络，第二阶段则致力于插值帧生成以实现场景与物体的平滑运动。我们比较了多种用于关键帧生成的时间条件处理方法。结果表明，在反映视频生成质量与人类偏好的各项指标上，采用独立时间块的设计优于时间层方案。此外，与其它掩码帧插值方法相比，我们设计的插值模型显著降低了计算成本。进一步地，我们评估了基于MoVQ的视频解码方案的不同配置，以提升一致性并实现更高的PSNR、SSIM、MSE及LPIPS分数。最终，我们将本流程与现有解决方案进行对比，在总体评分中位列第二，在开源方案中位列第一：CLIPSIM = 0.2976，FVD = 433.054。项目主页：https://ai-forever.github.io/kandinsky-video/