As text-to-image (T2I) synthesis models increase in size, they demand higher inference costs due to the need for more expensive GPUs with larger memory, which makes it challenging to reproduce these models in addition to the restricted access to training datasets. Our study aims to reduce these inference costs and explores how far the generative capabilities of T2I models can be extended using only publicly available datasets and open-source models. To this end, by using the de facto standard text-to-image model, Stable Diffusion XL (SDXL), we present three key practices in building an efficient T2I model: (1) Knowledge distillation: we explore how to effectively distill the generation capability of SDXL into an efficient U-Net and find that self-attention is the most crucial part. (2) Data: despite fewer samples, high-resolution images with rich captions are more crucial than a larger number of low-resolution images with short captions. (3) Teacher: Step-distilled Teacher allows T2I models to reduce the noising steps. Based on these findings, we build two types of efficient text-to-image models, called KOALA-Turbo &-Lightning, with two compact U-Nets (1B & 700M), reducing the model size up to 54% and 69% of the SDXL U-Net. In particular, the KOALA-Lightning-700M is 4x faster than SDXL while still maintaining satisfactory generation quality. Moreover, unlike SDXL, our KOALA models can generate 1024px high-resolution images on consumer-grade GPUs with 8GB of VRAMs (3060Ti). We believe that our KOALA models will have a significant practical impact, serving as cost-effective alternatives to SDXL for academic researchers and general users in resource-constrained environments.

翻译：随着文本到图像（T2I）合成模型规模的增大，其对推理成本的要求也日益提高，这主要源于需要配备更大内存的昂贵GPU。除了训练数据集的获取受限外，这也使得复现这些模型变得颇具挑战。本研究旨在降低这些推理成本，并探索仅使用公开可用的数据集和开源模型能将T2I模型的生成能力扩展到何种程度。为此，我们以事实上的标准文本到图像模型——Stable Diffusion XL（SDXL）为基础，提出了构建高效T2I模型的三个关键实践：（1）知识蒸馏：我们探索了如何将SDXL的生成能力有效蒸馏到一个高效的U-Net中，并发现自注意力机制是最关键的部分。（2）数据：尽管样本数量较少，但具有丰富描述的高分辨率图像比大量带有简短描述的低分辨率图像更为关键。（3）教师模型：采用逐步蒸馏的教师模型使得T2I模型能够减少噪声添加步骤。基于这些发现，我们构建了两种高效文本到图像模型，分别命名为KOALA-Turbo和KOALA-Lightning，它们采用两种紧凑型U-Net（10亿参数和7亿参数），将模型大小分别缩减至SDXL U-Net的54%和69%。特别地，KOALA-Lightning-700M的生成速度比SDXL快4倍，同时仍能保持令人满意的生成质量。此外，与SDXL不同，我们的KOALA模型能够在配备8GB显存（3060Ti）的消费级GPU上生成1024像素的高分辨率图像。我们相信，我们的KOALA模型将产生重大的实际影响，为资源受限环境下的学术研究者和普通用户提供具有成本效益的SDXL替代方案。