Recent advancements in text-to-image models have significantly enhanced image generation capabilities, yet a notable gap of open-source models persists in bilingual or Chinese language support. To address this need, we present Taiyi-Diffusion-XL, a new Chinese and English bilingual text-to-image model which is developed by extending the capabilities of CLIP and Stable-Diffusion-XL through a process of bilingual continuous pre-training. This approach includes the efficient expansion of vocabulary by integrating the most frequently used Chinese characters into CLIP's tokenizer and embedding layers, coupled with an absolute position encoding expansion. Additionally, we enrich text prompts by large vision-language model, leading to better images captions and possess higher visual quality. These enhancements are subsequently applied to downstream text-to-image models. Our empirical results indicate that the developed CLIP model excels in bilingual image-text retrieval.Furthermore, the bilingual image generation capabilities of Taiyi-Diffusion-XL surpass previous models. This research leads to the development and open-sourcing of the Taiyi-Diffusion-XL model, representing a notable advancement in the field of image generation, particularly for Chinese language applications. This contribution is a step forward in addressing the need for more diverse language support in multimodal research. The model and demonstration are made publicly available at \href{https://huggingface.co/IDEA-CCNL/Taiyi-Stable-Diffusion-XL-3.5B/}{this https URL}, fostering further research and collaboration in this domain.

翻译：当前文本到图像模型的进展显著提升了图像生成能力，但在开源模型中仍存在对双语或中文支持不足的显著缺口。为应对这一需求，我们提出了Taiyi-Diffusion-XL——一种全新的中英双语文本到图像模型，该模型通过双语连续预训练扩展CLIP与Stable-Diffusion-XL的能力而开发。该方法通过集成最常用的中文字符至CLIP的词元切分器与嵌入层实现高效的词汇扩展，并辅以绝对位置编码扩展。此外，我们利用大规模视觉语言模型丰富文本提示词，从而生成更优的图像描述并呈现更高视觉质量。这些改进随后应用于下游文本到图像模型。实验结果表明，所开发的CLIP模型在双语图文检索中表现出色。进一步地，Taiyi-Diffusion-XL的双语图像生成能力超越了先前模型。本研究最终开发并开源了Taiyi-Diffusion-XL模型，标志着图像生成领域（尤其是面向中文应用场景）的重要进步。此项贡献是为多模态研究中更广泛的语言支持需求向前迈出的一步。模型及演示已在\href{https://huggingface.co/IDEA-CCNL/Taiyi-Stable-Diffusion-XL-3.5B/}{此https链接}公开，以促进该领域的进一步研究与协作。