Video generation has seen remarkable progresses thanks to advancements in generative deep learning. Generated videos should not only display coherent and continuous movement but also meaningful movement in successions of scenes. Generating models such as Generative Adversarial Networks (GANs) or Variational Autoencoders (VAEs) and more recently Diffusion Networks have been used for generating short video sequences, usually of up to 16 frames. In this paper, we first propose a new type of video generator by enabling adversarial-based unconditional video generators with a variational encoder, akin to a VAE-GAN hybrid structure, in order to enable the generation process with inference capabilities. The proposed model, as in other video deep learning-based processing frameworks, incorporates two processing branches, one for content and another for movement. However, existing models struggle with the temporal scaling of the generated videos. In classical approaches when aiming to increase the generated video length, the resulting video quality degrades, particularly when considering generating significantly long sequences. To overcome this limitation, our research study extends the initially proposed VAE-GAN video generation model by employing a novel, memory-efficient approach to generate long videos composed of hundreds or thousands of frames ensuring their temporal continuity, consistency and dynamics. Our approach leverages a Markov chain framework with a recall mechanism, with each state representing a VAE-GAN short-length video generator. This setup allows for the sequential connection of generated video sub-sequences, enabling temporal dependencies, resulting in meaningful long video sequences.

翻译：得益于生成式深度学习的进步，视频生成领域已取得显著进展。生成的视频不仅应呈现连贯且连续的运动，还应在场景序列中展现有意义的动态变化。诸如生成对抗网络（GANs）或变分自编码器（VAEs）等生成模型，以及近期兴起的扩散网络，已被用于生成短时视频序列（通常最多16帧）。本文首先提出一种新型视频生成器，通过为基于对抗的无条件视频生成器配备变分编码器（类似于VAE-GAN混合结构），使生成过程具备推理能力。与其它基于深度学习的视频处理框架类似，所提模型包含两个处理分支：一个用于内容生成，另一个用于运动建模。然而，现有模型在生成视频的时间尺度扩展方面存在困难。传统方法在试图增加生成视频长度时，会导致视频质量下降，特别是在生成极长序列时尤为明显。为克服这一局限，本研究通过采用一种新颖的内存高效方法，扩展了最初提出的VAE-GAN视频生成模型，能够生成由数百或数千帧组成的长视频，同时确保其时间连续性、一致性与动态特性。我们的方法利用带有记忆机制的马尔可夫链框架，其中每个状态代表一个VAE-GAN短时视频生成器。该架构通过顺序连接生成的视频子序列，实现了时间依赖性建模，从而产生具有语义关联的长视频序列。