Diffusion-model-based text-guided image generation has recently made astounding progress, producing fascinating results in open-domain image manipulation tasks. Few models, however, currently have complete zero-shot capabilities for both global and local image editing due to the complexity and diversity of image manipulation tasks. In this work, we propose a method with a mixture-of-expert (MOE) controllers to align the text-guided capacity of diffusion models with different kinds of human instructions, enabling our model to handle various open-domain image manipulation tasks with natural language instructions. First, we use large language models (ChatGPT) and conditional image synthesis models (ControlNet) to generate a large number of global image transfer dataset in addition to the instruction-based local image editing dataset. Then, using an MOE technique and task-specific adaptation training on a large-scale dataset, our conditional diffusion model can edit images globally and locally. Extensive experiments demonstrate that our approach performs surprisingly well on various image manipulation tasks when dealing with open-domain images and arbitrary human instructions. Please refer to our project page: [https://oppo-mente-lab.github.io/moe_controller/]

翻译：基于扩散模型的文本引导图像生成近期取得了惊人的进展，在开放域图像操控任务中展现出令人瞩目的成果。然而，由于图像操控任务的复杂性和多样性，目前仅有少数模型具备同时完成全局与局部图像编辑的完整零样本能力。本研究提出一种基于混合专家（MoE）控制器的方法，用于对齐扩散模型的文本引导能力与不同类型的人类指令，使模型能够通过自然语言指令处理各类开放域图像操控任务。首先，我们利用大型语言模型（ChatGPT）和条件图像合成模型（ControlNet），除基于指令的局部图像编辑数据集外，额外生成大量全局图像迁移数据集。随后，通过MoE技术在大规模数据集上进行任务特定适配训练，我们的条件扩散模型能够实现全局与局部图像编辑。大量实验表明，在处理开放域图像与任意人类指令时，本方法在各类图像操控任务上表现优异。请参阅我们的项目页面：[https://oppo-mente-lab.github.io/moe_controller/]