As the development of large-scale Generative AI models evolve beyond text (1D) generation to include image (2D) and video (3D) generation, processing spatial and temporal information presents unique challenges to quality, performance, and efficiency. We present the first work towards understanding this new system design space for multi-modal text-to-image (TTI) and text-to-video (TTV) generation models. Current model architecture designs are bifurcated into 2 categories: Diffusion- and Transformer-based models. Our systematic performance characterization on a suite of eight representative TTI/TTV models shows that after state-of-the-art optimization techniques such as Flash Attention are applied, Convolution accounts for up to 44% of execution time for Diffusion-based TTI models, while Linear layers consume up to 49% of execution time for Transformer-based models. We additionally observe that Diffusion-based TTI models resemble the Prefill stage of LLM inference, and benefit from 1.1-2.5x greater speedup from Flash Attention than Transformer-based TTI models that resemble the Decode phase. Since optimizations designed for LLMs do not map directly onto TTI/TTV models, we must conduct a thorough characterization of these workloads to gain insights for new optimization opportunities. In doing so, we define sequence length in the context of TTI/TTV models and observe sequence length can vary up to 4x in Diffusion model inference. We additionally observe temporal aspects of TTV workloads pose unique system bottlenecks, with Temporal Attention accounting for over 60% of total Attention time. Overall, our in-depth system performance characterization is a critical first step towards designing efficient and deployable systems for emerging TTI/TTV workloads.

翻译：随着大规模生成式人工智能模型的发展从一维文本生成扩展到二维图像和三维视频生成，处理空间与时间信息对质量、性能和效率提出了独特挑战。本文首次致力于理解面向多模态文本到图像（TTI）与文本到视频（TTV）生成模型的新型系统设计空间。当前模型架构设计分为扩散模型与Transformer模型两类。我们对八种代表性TTI/TTV模型的系统性性能表征显示，在应用Flash Attention等最新优化技术后，扩散型TTI模型中卷积操作仍占执行时间的44%，而Transformer模型中的线性层消耗高达49%的执行时间。我们进一步观察到，扩散型TTI模型类似于大语言模型推理中的预填充阶段，其从Flash Attention获得的加速效果（1.1-2.5倍）优于类似解码阶段的Transformer型TTI模型。由于面向大语言模型的优化无法直接迁移至TTI/TTV模型，必须对这些工作负载进行深入表征以发掘新的优化机遇。为此，我们在TTI/TTV模型语境下定义了序列长度，并发现扩散模型推理中序列长度变化可达4倍。此外，我们注意到TTV工作负载的时间维度特性带来独特系统瓶颈，其中时间注意力占总注意力时间的60%以上。总体而言，本研究的深度系统性能表征是为新兴TTI/TTV工作负载设计高效可部署系统的关键第一步。