Advancements in language foundation models have primarily fueled the recent surge in artificial intelligence. In contrast, generative learning of non-textual modalities, especially videos, significantly trails behind language modeling. This thesis chronicles our endeavor to build multi-task models for generating videos and other modalities under diverse conditions, as well as for understanding and compression applications. Given the high dimensionality of visual data, we pursue concise and accurate latent representations. Our video-native spatial-temporal tokenizers preserve high fidelity. We unveil a novel approach to mapping bidirectionally between visual observation and interpretable lexical terms. Furthermore, our scalable visual token representation proves beneficial across generation, compression, and understanding tasks. This achievement marks the first instances of language models surpassing diffusion models in visual synthesis and a video tokenizer outperforming industry-standard codecs. Within these multi-modal latent spaces, we study the design of multi-task generative models. Our masked multi-task transformer excels at the quality, efficiency, and flexibility of video generation. We enable a frozen language model, trained solely on text, to generate visual content. Finally, we build a scalable generative multi-modal transformer trained from scratch, enabling the generation of videos containing high-fidelity motion with the corresponding audio given diverse conditions. Throughout the course, we have shown the effectiveness of integrating multiple tasks, crafting high-fidelity latent representation, and generating multiple modalities. This work suggests intriguing potential for future exploration in generating non-textual data and enabling real-time, interactive experiences across various media forms.

翻译：语言基础模型的进展主要推动了近期人工智能的蓬勃发展。相比之下，非文本模态（尤其是视频）的生成式学习显著落后于语言建模。本论文系统阐述了我们在多样化条件下构建用于生成视频及其他模态、以及用于理解与压缩应用的多任务模型的探索历程。鉴于视觉数据的高维特性，我们致力于寻求简洁而准确的潜在表示。我们提出的视频原生时空分词器保持了高保真度。我们揭示了一种在视觉观测与可解释词汇术语之间进行双向映射的新方法。此外，我们提出的可扩展视觉分词表示在生成、压缩和理解任务中均被证明具有优势。这一成就标志着语言模型在视觉合成方面首次超越扩散模型，以及视频分词器性能优于行业标准编解码器。在这些多模态潜在空间中，我们研究了多任务生成模型的设计。我们提出的掩码多任务Transformer在视频生成的质量、效率和灵活性方面表现出色。我们使一个仅基于文本训练的冻结语言模型能够生成视觉内容。最后，我们构建了一个从头开始训练的可扩展生成式多模态Transformer，能够在多样化条件下生成包含高保真度运动及对应音频的视频。在整个研究过程中，我们展示了整合多任务、构建高保真潜在表示以及生成多模态的有效性。这项工作为未来在非文本数据生成以及实现跨媒体形式的实时交互体验方面的探索揭示了引人入胜的潜力。