Recent successes suggest that an image can be manipulated by a text prompt, e.g., a landscape scene on a sunny day is manipulated into the same scene on a rainy day driven by a text input "raining". These approaches often utilize a StyleCLIP-based image generator, which leverages multi-modal (text and image) embedding space. However, we observe that such text inputs are often bottlenecked in providing and synthesizing rich semantic cues, e.g., differentiating heavy rain from rain with thunderstorms. To address this issue, we advocate leveraging an additional modality, sound, which has notable advantages in image manipulation as it can convey more diverse semantic cues (vivid emotions or dynamic expressions of the natural world) than texts. In this paper, we propose a novel approach that first extends the image-text joint embedding space with sound and applies a direct latent optimization method to manipulate a given image based on audio input, e.g., the sound of rain. Our extensive experiments show that our sound-guided image manipulation approach produces semantically and visually more plausible manipulation results than the state-of-the-art text and sound-guided image manipulation methods, which are further confirmed by our human evaluations. Our downstream task evaluations also show that our learned image-text-sound joint embedding space effectively encodes sound inputs.

翻译：近期研究进展表明，图像可通过文本提示进行编辑，例如，通过输入文本“下雨”将晴天风景图转换为同一场景的雨天图像。这些方法通常利用基于StyleCLIP的图像生成器，借助多模态（文本与图像）嵌入空间。然而，我们发现此类文本输入在提供和合成丰富语义线索时存在瓶颈，例如难以区分暴雨与雷阵雨。为解决此问题，我们主张利用另一种模态——声音，其在图像编辑中具有显著优势，能够比文本传递更多样化的语义线索（如自然界的生动情感或动态表达）。本文提出一种新颖方法，首先将图像-文本联合嵌入空间扩展至声音模态，并采用直接潜在优化方法，基于音频输入（如雨声）编辑给定图像。大量实验表明，我们的声音引导图像编辑方法在语义和视觉上均比当前最先进的文本及声音引导图像编辑方法生成更合理的结果，这通过人工评估进一步得到验证。下游任务评估也显示，我们学习的图像-文本-声音联合嵌入空间能有效编码声音输入。