We introduce a novel method to automatically generate an artistic typography by stylizing one or more letter fonts to visually convey the semantics of an input word, while ensuring that the output remains readable. To address an assortment of challenges with our task at hand including conflicting goals (artistic stylization vs. legibility), lack of ground truth, and immense search space, our approach utilizes large language models to bridge texts and visual images for stylization and build an unsupervised generative model with a diffusion model backbone. Specifically, we employ the denoising generator in Latent Diffusion Model (LDM), with the key addition of a CNN-based discriminator to adapt the input style onto the input text. The discriminator uses rasterized images of a given letter/word font as real samples and output of the denoising generator as fake samples. Our model is coined DS-Fusion for discriminated and stylized diffusion. We showcase the quality and versatility of our method through numerous examples, qualitative and quantitative evaluation, as well as ablation studies. User studies comparing to strong baselines including CLIPDraw and DALL-E 2, as well as artist-crafted typographies, demonstrate strong performance of DS-Fusion.

翻译：我们提出一种新颖方法，通过将一个或多个字母字体风格化以在视觉上传达输入词语的语义，同时确保输出结果的可读性，从而自动生成艺术字体。针对任务中存在的多重挑战——包括冲突目标（艺术风格化与可读性）、缺少真值以及巨大的搜索空间——我们的方法利用大型语言模型桥接文本与视觉图像以完成风格化，并构建基于扩散模型骨干的无监督生成模型。具体而言，我们采用潜在扩散模型（LDM）中的去噪生成器，关键创新在于引入基于CNN的判别器，将输入风格适配到输入文本上。该判别器以给定字母/词语字体的光栅化图像作为真实样本，以去噪生成器的输出作为伪造样本。我们将模型命名为DS-Fusion（判别式与风格化扩散）。通过大量示例、定性与定量评估以及消融实验，我们展示了该方法的质量与多样性。与包括CLIPDraw和DALL-E 2在内的强基线方法以及艺术家手工制作字体的用户对比研究表明，DS-Fusion表现出色。