Recently, text-to-image denoising diffusion probabilistic models (DDPMs) have demonstrated impressive image generation capabilities and have also been successfully applied to image inpainting. However, in practice, users often require more control over the inpainting process beyond textual guidance, especially when they want to composite objects with customized appearance, color, shape, and layout. Unfortunately, existing diffusion-based inpainting methods are limited to single-modal guidance and require task-specific training, hindering their cross-modal scalability. To address these limitations, we propose Uni-paint, a unified framework for multimodal inpainting that offers various modes of guidance, including unconditional, text-driven, stroke-driven, exemplar-driven inpainting, as well as a combination of these modes. Furthermore, our Uni-paint is based on pretrained Stable Diffusion and does not require task-specific training on specific datasets, enabling few-shot generalizability to customized images. We have conducted extensive qualitative and quantitative evaluations that show our approach achieves comparable results to existing single-modal methods while offering multimodal inpainting capabilities not available in other methods. Code will be available at https://github.com/ysy31415/unipaint.

翻译：近期，文本到图像的去噪扩散概率模型（DDPMs）在图像生成任务中展现了卓越能力，并已成功应用于图像修复领域。然而在实际应用中，用户往往需要超越文本引导的更多控制手段，特别是当需要合成具有定制外观、颜色、形状和布局的物体时。遗憾的是，现有基于扩散模型的修复方法仅支持单模态引导，且需要针对特定任务进行训练，这限制了其跨模态扩展能力。为解决上述问题，我们提出Uni-paint——一个统一的多模态修复框架，支持无条件修复、文本驱动修复、笔画驱动修复、示例驱动修复及其组合模式。此外，本框架基于预训练的Stable Diffusion构建，无需在特定数据集上进行任务专用训练，即可实现对定制化图像的少样本泛化。通过大量定性与定量评估，我们证明了该方法在保持与现有单模态方法相当性能的同时，实现了其他方法所不具备的多模态修复能力。代码将开源至https://github.com/ysy31415/unipaint。