Diffusion model based Text-to-Image has achieved impressive achievements recently. Although current technology for synthesizing images is highly advanced and capable of generating images with high fidelity, it is still possible to give the show away when focusing on the text area in the generated image. To address this issue, we introduce AnyText, a diffusion-based multilingual visual text generation and editing model, that focuses on rendering accurate and coherent text in the image. AnyText comprises a diffusion pipeline with two primary elements: an auxiliary latent module and a text embedding module. The former uses inputs like text glyph, position, and masked image to generate latent features for text generation or editing. The latter employs an OCR model for encoding stroke data as embeddings, which blend with image caption embeddings from the tokenizer to generate texts that seamlessly integrate with the background. We employed text-control diffusion loss and text perceptual loss for training to further enhance writing accuracy. AnyText can write characters in multiple languages, to the best of our knowledge, this is the first work to address multilingual visual text generation. It is worth mentioning that AnyText can be plugged into existing diffusion models from the community for rendering or editing text accurately. After conducting extensive evaluation experiments, our method has outperformed all other approaches by a significant margin. Additionally, we contribute the first large-scale multilingual text images dataset, AnyWord-3M, containing 3 million image-text pairs with OCR annotations in multiple languages. Based on AnyWord-3M dataset, we propose AnyText-benchmark for the evaluation of visual text generation accuracy and quality. Our project will be open-sourced on https://github.com/tyxsspa/AnyText to improve and promote the development of text generation technology.

翻译：基于扩散模型的文本到图像技术近期取得了令人瞩目的进展。尽管当前图像合成技术高度成熟，能够生成高保真度的图像，但在聚焦生成图像中的文字区域时，仍可能出现破绽。为解决这一问题，我们提出AnyText——一个基于扩散的多语言视觉文字生成与编辑模型，专注于在图像中呈现准确且连贯的文字。AnyText包含一个扩散流程，由两个核心组件构成：辅助潜在模块和文本嵌入模块。前者利用文字字形、位置和掩码图像等输入，为文字生成或编辑生成潜在特征；后者采用OCR模型将笔画数据编码为嵌入向量，与来自分词器的图像描述嵌入相融合，从而生成与背景无缝整合的文字。我们采用文字控制扩散损失和文字感知损失进行训练，以进一步提升书写准确性。AnyText支持多种语言字符，据我们所知，这是首个解决多语言视觉文字生成问题的研究。值得一提的是，AnyText可直接集成至社区现有的扩散模型中，用于精确的文字渲染或编辑。经过大量评估实验，我们的方法以显著优势超越了所有现有方法。此外，我们贡献了首个大规模多语言文字图像数据集AnyWord-3M，包含300万对图像-文本数据及多语言的OCR标注。基于AnyWord-3M数据集，我们提出AnyText-benchmark基准，用于评估视觉文字生成的准确性和质量。我们的项目将在https://github.com/tyxsspa/AnyText开源，以推动文字生成技术的发展。