Diffusion models are the de facto approach for generating high-quality images and videos, but learning high-dimensional models remains a formidable task due to computational and optimization challenges. Existing methods often resort to training cascaded models in pixel space or using a downsampled latent space of a separately trained auto-encoder. In this paper, we introduce Matryoshka Diffusion Models(MDM), an end-to-end framework for high-resolution image and video synthesis. We propose a diffusion process that denoises inputs at multiple resolutions jointly and uses a NestedUNet architecture where features and parameters for small-scale inputs are nested within those of large scales. In addition, MDM enables a progressive training schedule from lower to higher resolutions, which leads to significant improvements in optimization for high-resolution generation. We demonstrate the effectiveness of our approach on various benchmarks, including class-conditioned image generation, high-resolution text-to-image, and text-to-video applications. Remarkably, we can train a single pixel-space model at resolutions of up to 1024x1024 pixels, demonstrating strong zero-shot generalization using the CC12M dataset, which contains only 12 million images.

翻译：扩散模型是生成高质量图像和视频的事实标准方法，但由于计算和优化挑战，学习高维模型仍是一项艰巨任务。现有方法通常采用在像素空间中训练级联模型，或利用单独训练的自编码器所生成的下采样潜在空间。本文提出套娃扩散模型（MDM），一种用于高分辨率图像和视频合成的端到端框架。我们提出一种扩散过程，该过程联合对多分辨率输入进行去噪，并采用嵌套UNet架构，其中小尺度输入的特征和参数嵌套在大尺度输入的特征和参数中。此外，MDM支持从低到高分辨率的渐进式训练策略，从而显著改善高分辨率生成的优化效果。我们在多个基准上验证了该方法的有效性，包括类别条件图像生成、高分辨率文本到图像以及文本到视频应用。值得注意的是，我们能够训练一个分辨率高达1024×1024像素的单一像素空间模型，在使用仅包含1200万图像的CC12M数据集时展示了强大的零样本泛化能力。