Text-to-Image generation (TTI) technologies are advancing rapidly, especially in the English language communities. However, English-native TTI models inherently carry biases from English world centric training data, which creates a dilemma for development of other language-native TTI models. One common choice is fine-tuning the English-native TTI model with translated samples from non-English communities. It falls short of fully addressing the model bias problem. Alternatively, training non-English language native models from scratch can effectively resolve the English world bias, but diverges from the English TTI communities, thus not able to utilize the strides continuously gaining in the English TTI communities any more. To build non-English language native TTI model meanwhile keep compatability with the English TTI communities, we propose a novel model structure referred as "Bridge Diffusion Model" (BDM). The proposed BDM employs a backbone-branch network structure to learn the non-English language semantics while keep the latent space compatible with the English-native TTI backbone, in an end-to-end manner. The unique advantages of the proposed BDM are that it's not only adept at generating images that precisely depict non-English language semantics, but also compatible with various English-native TTI plugins, such as different checkpoints, LoRA, ControlNet, Dreambooth, and Textual Inversion, etc. Moreover, BDM can concurrently generate content seamlessly combining both non-English native and English-native semantics within a single image, fostering cultural interaction. We verify our method by applying BDM to build a Chinese-native TTI model, whereas the method is generic and applicable to any other language.
翻译:文本到图像生成技术在英语社区中发展迅速。然而,英语原生TTI模型天然携带以英语世界为中心的训练数据所导致的偏见,这为非英语原生TTI模型的发展带来了困境。常见的做法是利用非英语社区的翻译样本对英语原生TTI模型进行微调,但这未能充分解决模型偏见问题。另一种方案是从零开始训练非英语原生TTI模型,这能有效消除英语世界偏见,但会脱离英语TTI社区,从而无法继续利用英语TTI社区持续取得的进展。为构建非英语原生TTI模型并同时保持与英语TTI社区的兼容性,我们提出一种名为“桥接扩散模型”的新颖模型结构。该模型采用主干-分支网络结构,以端到端方式学习非英语语言语义,同时保持潜空间与英语原生TTI主干兼容。BDM的独特优势在于:不仅能精确生成描述非英语语言语义的图像,还与多种英语原生TTI插件兼容,如不同检查点、LoRA、ControlNet、Dreambooth和Textual Inversion等。此外,BDM可在单张图像中无缝融合非英语原生语义与英语原生语义,促进文化交互。我们通过将BDM应用于构建中文原生TTI模型来验证该方法,但该方法具有通用性,可适用于任何其他语言。