The field of generative image inpainting and object insertion has made significant progress with the recent advent of latent diffusion models. Utilizing a precise object mask can greatly enhance these applications. However, due to the challenges users encounter in creating high-fidelity masks, there is a tendency for these methods to rely on more coarse masks (e.g., bounding box) for these applications. This results in limited control and compromised background content preservation. To overcome these limitations, we introduce SmartMask, which allows any novice user to create detailed masks for precise object insertion. Combined with a ControlNet-Inpaint model, our experiments demonstrate that SmartMask achieves superior object insertion quality, preserving the background content more effectively than previous methods. Notably, unlike prior works the proposed approach can also be used even without user-mask guidance, which allows it to perform mask-free object insertion at diverse positions and scales. Furthermore, we find that when used iteratively with a novel instruction-tuning based planning model, SmartMask can be used to design detailed layouts from scratch. As compared with user-scribble based layout design, we observe that SmartMask allows for better quality outputs with layout-to-image generation methods. Project page is available at https://smartmask-gen.github.io

翻译：生成式图像修复与物体插入领域随着近期潜在扩散模型的出现取得了显著进展。利用精确的物体掩码能够显著增强这些应用的效果。然而，由于用户在创建高保真掩码时面临挑战，这些方法更倾向于使用较粗糙的掩码（例如边界框）进行应用，导致控制能力受限并影响背景内容保留。为克服这些局限，我们提出SmartMask，它使得任何新手用户都能创建精细掩码以实现精确物体插入。结合ControlNet-Inpainting模型，实验表明SmartMask在插入质量上优于现有方法，且能更有效地保留背景内容。值得注意的是，与先前工作不同，本方法无需用户掩码引导也可使用，从而能在不同位置和尺度上进行无掩码物体插入。此外，我们发现当与基于指令调优的新型规划模型迭代结合时，SmartMask可从零开始设计精细布局。与基于用户涂鸦的布局设计相比，SmartMask结合布局到图像生成方法可获得更高质量输出。项目页面见https://smartmask-gen.github.io