Direct speech-to-image generation has recently shown promising results. However, compared to text-to-image generation, there is still a large gap to enclose. Current approaches use two stages to tackle this task: speech encoding network and image generative adversarial network (GAN). The speech encoding networks in these approaches produce embeddings that do not capture sufficient linguistic information to semantically represent the input speech. GANs suffer from issues such as non-convergence, mode collapse, and diminished gradient, which result in unstable model parameters, limited sample diversity, and ineffective generator learning, respectively. To address these weaknesses, we introduce a framework called Speak the Art (STA) which consists of a speech encoding network and a VQ-Diffusion network conditioned on speech embeddings. To improve speech embeddings, the speech encoding network is supervised by a large pre-trained image-text model during training. Replacing GANs with diffusion leads to more stable training and the generation of diverse images. Additionally, we investigate the feasibility of extending our framework to be multilingual. As a proof of concept, we trained our framework with two languages: English and Arabic. Finally, we show that our results surpass state-of-the-art models by a large margin.

翻译：直接语音到图像生成技术近期已展现出有前景的结果。然而，与文本到图像生成相比，仍存在较大差距需要弥合。当前方法采用两阶段处理此任务：语音编码网络与图像生成对抗网络（GAN）。这些方法中的语音编码网络产生的嵌入向量未能捕获足够的语言信息以语义化表征输入语音。GAN则受限于非收敛、模式崩溃及梯度消失等问题，分别导致模型参数不稳定、样本多样性受限以及生成器学习低效。为克服这些不足，我们提出一个名为“语音成画”（STA）的框架，该框架包含一个语音编码网络和一个以语音嵌入为条件的VQ-Diffusion网络。为提升语音嵌入质量，语音编码网络在训练过程中受到一个大型预训练图像-文本模型的监督。用扩散模型替代GAN可实现更稳定的训练并生成多样化的图像。此外，我们探究了将该框架扩展至多语言场景的可行性。作为概念验证，我们使用英语和阿拉伯语两种语言训练了该框架。最终，实验结果表明我们的方法大幅超越了现有最优模型。