We present InstructDiffusion, a unifying and generic framework for aligning computer vision tasks with human instructions. Unlike existing approaches that integrate prior knowledge and pre-define the output space (e.g., categories and coordinates) for each vision task, we cast diverse vision tasks into a human-intuitive image-manipulating process whose output space is a flexible and interactive pixel space. Concretely, the model is built upon the diffusion process and is trained to predict pixels according to user instructions, such as encircling the man's left shoulder in red or applying a blue mask to the left car. InstructDiffusion could handle a variety of vision tasks, including understanding tasks (such as segmentation and keypoint detection) and generative tasks (such as editing and enhancement). It even exhibits the ability to handle unseen tasks and outperforms prior methods on novel datasets. This represents a significant step towards a generalist modeling interface for vision tasks, advancing artificial general intelligence in the field of computer vision.

翻译：我们提出InstructDiffusion，一个统一且通用的框架，用于将计算机视觉任务与人类指令对齐。不同于现有方法为每个视觉任务整合先验知识并预定义输出空间（如类别和坐标），我们将多样化视觉任务转化为一种符合人类直觉的图像操作过程，其输出空间为灵活且交互式的像素空间。具体而言，该模型基于扩散过程构建，并通过训练根据用户指令预测像素值，例如用红色圈出男性左肩或为左侧汽车应用蓝色掩码。InstructDiffusion能够处理多种视觉任务，包括理解型任务（如分割和关键点检测）和生成型任务（如编辑与增强）。它甚至展现出处理未见任务的能力，并在新数据集上优于现有方法。这标志着迈向视觉任务通用建模接口的重要一步，推动了计算机视觉领域的人工通用智能发展。