We introduce a novel diffusion transformer, LazyDiffusion, that generates partial image updates efficiently. Our approach targets interactive image editing applications in which, starting from a blank canvas or an image, a user specifies a sequence of localized image modifications using binary masks and text prompts. Our generator operates in two phases. First, a context encoder processes the current canvas and user mask to produce a compact global context tailored to the region to generate. Second, conditioned on this context, a diffusion-based transformer decoder synthesizes the masked pixels in a "lazy" fashion, i.e., it only generates the masked region. This contrasts with previous works that either regenerate the full canvas, wasting time and computation, or confine processing to a tight rectangular crop around the mask, ignoring the global image context altogether. Our decoder's runtime scales with the mask size, which is typically small, while our encoder introduces negligible overhead. We demonstrate that our approach is competitive with state-of-the-art inpainting methods in terms of quality and fidelity while providing a 10x speedup for typical user interactions, where the editing mask represents 10% of the image.

翻译：我们提出一种新颖的扩散变换器——LazyDiffusion，它能高效地生成图像的局部更新。该方法面向交互式图像编辑应用，用户可以从空白画布或现有图像出发，通过二进制掩码和文本提示指定一系列局部图像修改。我们的生成器分为两个阶段：首先，上下文编码器处理当前画布和用户掩码，生成针对待生成区域的紧凑全局上下文；其次，基于此上下文，基于扩散的变换器解码器以“懒惰”方式合成掩码像素，即仅生成掩码区域。这与先前的方法形成对比：先前方法要么重新生成整个画布，浪费时间和计算资源，要么将处理限制在掩码周围的紧密矩形裁剪区域，完全忽略全局图像上下文。我们的解码器运行时消耗与掩码大小（通常较小）成线性比例，而编码器引入的计算开销可忽略不计。我们证明，该方法在质量和保真度方面能与最先进图像修复方法相媲美，同时为典型用户交互（当编辑掩码占图像面积的10%时）提供10倍速度提升。