Autoregressive transformers have shown remarkable success in video generation. However, the transformers are prohibited from directly learning the long-term dependency in videos due to the quadratic complexity of self-attention, and inherently suffering from slow inference time and error propagation due to the autoregressive process. In this paper, we propose Memory-efficient Bidirectional Transformer (MeBT) for end-to-end learning of long-term dependency in videos and fast inference. Based on recent advances in bidirectional transformers, our method learns to decode the entire spatio-temporal volume of a video in parallel from partially observed patches. The proposed transformer achieves a linear time complexity in both encoding and decoding, by projecting observable context tokens into a fixed number of latent tokens and conditioning them to decode the masked tokens through the cross-attention. Empowered by linear complexity and bidirectional modeling, our method demonstrates significant improvement over the autoregressive Transformers for generating moderately long videos in both quality and speed.

翻译：自回归Transformer在视频生成领域取得了显著成功。然而，由于自注意力机制的二次复杂度，这些Transformer无法直接学习视频中的长期依赖关系，且固有地受限于自回归过程导致的推理速度慢和误差传播问题。本文提出记忆高效双向Transformer（MeBT），用于端到端学习视频中的长期依赖并实现快速推理。基于双向Transformer的最新进展，我们的方法能够从部分观测的补丁中并行解码视频的完整时空体。通过将可观测的上下文标记投影为固定数量的潜在标记，并利用交叉注意力机制将其作为条件解码被掩蔽的标记，所提出的Transformer在编码和解码阶段均实现了线性时间复杂度。得益于线性复杂度和双向建模能力，我们的方法在生成中等长度视频的质量和速度上均显著优于自回归Transformer。