Recent advancements in text-to-image models have significantly enhanced image generation capabilities, yet a notable gap of open-source models persists in bilingual or Chinese language support. To address this need, we present Taiyi-Diffusion-XL, a new Chinese and English bilingual text-to-image model which is developed by extending the capabilities of CLIP and Stable-Diffusion-XL through a process of bilingual continuous pre-training. This approach includes the efficient expansion of vocabulary by integrating the most frequently used Chinese characters into CLIP's tokenizer and embedding layers, coupled with an absolute position encoding expansion. Additionally, we enrich text prompts by large vision-language model, leading to better images captions and possess higher visual quality. These enhancements are subsequently applied to downstream text-to-image models. Our empirical results indicate that the developed CLIP model excels in bilingual image-text retrieval.Furthermore, the bilingual image generation capabilities of Taiyi-Diffusion-XL surpass previous models. This research leads to the development and open-sourcing of the Taiyi-Diffusion-XL model, representing a notable advancement in the field of image generation, particularly for Chinese language applications. This contribution is a step forward in addressing the need for more diverse language support in multimodal research. The model and demonstration are made publicly available at \href{https://huggingface.co/IDEA-CCNL/Taiyi-Stable-Diffusion-XL-3.5B/}{this https URL}, fostering further research and collaboration in this domain.

翻译：近期文本到图像模型的进展显著提升了图像生成能力，但在双语或中文支持方面，开源模型仍存在明显不足。为满足这一需求，我们提出了太易-扩散-XL——一种新的中英双语文本到图像生成模型。该模型通过双语持续预训练过程，扩展了CLIP和Stable-Diffusion-XL的能力而开发。该方法通过将最常用的汉字集成到CLIP的分词器和嵌入层中，并结合绝对位置编码扩展，实现了词汇表的高效扩充。此外，我们利用大型视觉语言模型丰富文本提示，从而获得更好的图像描述和更高的视觉质量。这些增强技术随后被应用于下游文本到图像模型。实验结果表明，所开发的CLIP模型在双语图文检索任务中表现优异。进一步地，太易-扩散-XL的双语图像生成能力超越了先前模型。本研究推动了太易-扩散-XL模型的开发与开源，标志着图像生成领域特别是中文应用方向的重要进展。这一贡献为满足多模态研究中更多样化语言支持的需求迈出了关键一步。模型及演示已通过\href{https://huggingface.co/IDEA-CCNL/Taiyi-Stable-Diffusion-XL-3.5B/}{此链接}公开发布，以促进该领域的进一步研究与协作。