Despite the progress made in the style transfer task, most previous work focus on transferring only relatively simple features like color or texture, while missing more abstract concepts such as overall art expression or painter-specific traits. However, these abstract semantics can be captured by models like DALL-E or CLIP, which have been trained using huge datasets of images and textual documents. In this paper, we propose StylerDALLE, a style transfer method that exploits both of these models and uses natural language to describe abstract art styles. Specifically, we formulate the language-guided style transfer task as a non-autoregressive token sequence translation, i.e., from input content image to output stylized image, in the discrete latent space of a large-scale pretrained vector-quantized tokenizer, e.g., the discrete variational auto-encoder (dVAE) of DALL-E. To incorporate style information, we propose a Reinforcement Learning strategy with CLIP-based language supervision that ensures stylization and content preservation simultaneously. Experimental results demonstrate the superiority of our method, which can effectively transfer art styles using language instructions at different granularities. Code is available at https://github.com/zipengxuc/StylerDALLE.

翻译：尽管风格迁移任务取得了进展，但以往研究大多聚焦于迁移颜色或纹理等相对简单的特征，而忽略了整体艺术表现或画家特定风格等抽象概念。然而，DALL-E或CLIP等模型能够捕捉这些抽象语义，这些模型通过海量图像与文本数据集训练而成。本文提出StylerDALLE——一种同时利用上述两种模型、通过自然语言描述抽象艺术风格的迁移方法。具体而言，我们将语言引导风格迁移任务建模为大规模预训练向量量化分词器（如DALL-E中的离散变分自编码器dVAE）离散潜在空间中的非自回归令牌序列翻译，即从输入内容图像到输出风格化图像的转换。为融入风格信息，我们提出一种基于CLIP语言监督的强化学习策略，可同时确保风格化与内容保留。实验结果表明，该方法能有效通过不同粒度的语言指令迁移艺术风格，性能优于现有方法。代码已开源：https://github.com/zipengxuc/StylerDALLE。