We propose a zero-shot approach to image harmonization, aiming to overcome the reliance on large amounts of synthetic composite images in existing methods. These methods, while showing promising results, involve significant training expenses and often struggle with generalization to unseen images. To this end, we introduce a fully modularized framework inspired by human behavior. Leveraging the reasoning capabilities of recent foundation models in language and vision, our approach comprises three main stages. Initially, we employ a pretrained vision-language model (VLM) to generate descriptions for the composite image. Subsequently, these descriptions guide the foreground harmonization direction of a text-to-image generative model (T2I). We refine text embeddings for enhanced representation of imaging conditions and employ self-attention and edge maps for structure preservation. Following each harmonization iteration, an evaluator determines whether to conclude or modify the harmonization direction. The resulting framework, mirroring human behavior, achieves harmonious results without the need for extensive training. We present compelling visual results across diverse scenes and objects, along with a user study validating the effectiveness of our approach.

翻译：我们提出一种零样本图像和谐化方法，旨在克服现有方法对大量合成复合图像数据的依赖。现有方法虽取得显著成效，但训练成本高昂，且常难以泛化至未见图像。为此，我们借鉴人类行为模式，引入一个完全模块化的框架。该框架利用近期语言与视觉基础模型的推理能力，包含三个主要阶段：首先，使用预训练视觉语言模型（VLM）为复合图像生成描述；其次，这些描述引导文本到图像生成模型（T2I）的前景和谐化方向；我们优化文本嵌入以增强对成像条件的表征，并采用自注意力机制与边缘图实现结构保持。每次和谐化迭代后，评估器决定是否终止或调整和谐化方向。该框架模拟人类行为，无需大量训练即可实现和谐化效果。我们展示了涵盖多样场景与物体的视觉结果，并通过用户研究验证了方法的有效性。