Cross-Modal learning tasks have picked up pace in recent times. With plethora of applications in diverse areas, generation of novel content using multiple modalities of data has remained a challenging problem. To address the same, various generative modelling techniques have been proposed for specific tasks. Novel and creative image generation is one important aspect for industrial application which could help as an arm for novel content generation. Techniques proposed previously used Generative Adversarial Network(GAN), autoregressive models and Variational Autoencoders (VAE) for accomplishing similar tasks. These approaches are limited in their capability to produce images guided by either text instructions or rough sketch images decreasing the overall performance of image generator. We used state of the art diffusion models to generate creative art by primarily leveraging text with additional support of rough sketches. Diffusion starts with a pattern of random dots and slowly converts that pattern into a design image using the guiding information fed into the model. Diffusion models have recently outperformed other generative models in image generation tasks using cross modal data as guiding information. The initial experiments for this task of novel image generation demonstrated promising qualitative results.

翻译：跨模态学习任务近年来发展迅速。尽管在多个领域拥有众多应用，利用多模态数据生成新颖内容仍是一个具有挑战性的问题。为解决这一问题，针对特定任务提出了多种生成建模技术。新颖且富有创意的图像生成是工业应用的重要方面，可作为新颖内容生成的辅助手段。此前提出的技术采用生成对抗网络（GAN）、自回归模型和变分自编码器（VAE）来完成类似任务。这些方法在生成由文本指令或粗略草图引导的图像方面能力有限，降低了图像生成器的整体性能。我们采用最先进的扩散模型，主要通过文本引导，并辅以粗略草图的额外支持，来生成创意艺术。扩散过程从随机点阵模式开始，利用输入模型的引导信息，逐步将该模式转化为设计图像。扩散模型近来在使用跨模态数据作为引导信息的图像生成任务中，已超越其他生成模型。这项新颖图像生成任务的初期实验展示了令人鼓舞的定性结果。