Diffusion model based Text-to-Image has achieved impressive achievements recently. Although current technology for synthesizing images is highly advanced and capable of generating images with high fidelity, it is still possible to give the show away when focusing on the text area in the generated image. To address this issue, we introduce AnyText, a diffusion-based multilingual visual text generation and editing model, that focuses on rendering accurate and coherent text in the image. AnyText comprises a diffusion pipeline with two primary elements: an auxiliary latent module and a text embedding module. The former uses inputs like text glyph, position, and masked image to generate latent features for text generation or editing. The latter employs an OCR model for encoding stroke data as embeddings, which blend with image caption embeddings from the tokenizer to generate texts that seamlessly integrate with the background. We employed text-control diffusion loss and text perceptual loss for training to further enhance writing accuracy. AnyText can write characters in multiple languages, to the best of our knowledge, this is the first work to address multilingual visual text generation. It is worth mentioning that AnyText can be plugged into existing diffusion models from the community for rendering or editing text accurately. After conducting extensive evaluation experiments, our method has outperformed all other approaches by a significant margin. Additionally, we contribute the first large-scale multilingual text images dataset, AnyWord-3M, containing 3 million image-text pairs with OCR annotations in multiple languages. Based on AnyWord-3M dataset, we propose AnyText-benchmark for the evaluation of visual text generation accuracy and quality. Our project will be open-sourced on https://github.com/tyxsspa/AnyText to improve and promote the development of text generation technology.

翻译：基于扩散模型的文本到图像生成技术近期取得了显著进展。尽管当前图像合成技术高度发达，能够生成高保真度的图像，但在聚焦生成图像中的文本区域时，仍可能显露破绽。为解决此问题，我们提出了AnyText——一种基于扩散模型的多语言视觉文本生成与编辑模型，专注于在图像中渲染准确且连贯的文本。AnyText包含一个扩散流水线，其两大核心组件为：辅助潜变量模块和文本嵌入模块。前者利用文本字形、位置和掩码图像等输入生成用于文本生成或编辑的潜特征；后者采用OCR模型将笔画数据编码为嵌入向量，并与分词器生成的图像描述嵌入融合，生成与背景无缝融合的文本。我们采用文本控制扩散损失和文本感知损失进行训练，以进一步提升书写准确性。AnyText支持多语言文字生成，据我们所知，这是首项解决多语言视觉文本生成的工作。值得一提的是，AnyText可无缝集成至社区现有扩散模型中，用于精确渲染或编辑文本。经过大量评估实验，我们的方法以显著优势超越了所有现有方案。此外，我们贡献了首个大规模多语言文本图像数据集AnyWord-3M，包含300万对多语言图像-文本对及OCR标注。基于AnyWord-3M数据集，我们提出了用于评估视觉文本生成准确性与质量的AnyText-benchmark基准。本项目将在https://github.com/tyxsspa/AnyText开源，以推动文本生成技术的发展。