In this paper, we introduce a privacy-preserving stable diffusion framework leveraging homomorphic encryption, called HE-Diffusion, which primarily focuses on protecting the denoising phase of the diffusion process. HE-Diffusion is a tailored encryption framework specifically designed to align with the unique architecture of stable diffusion, ensuring both privacy and functionality. To address the inherent computational challenges, we propose a novel min-distortion method that enables efficient partial image encryption, significantly reducing the overhead without compromising the model's output quality. Furthermore, we adopt a sparse tensor representation to expedite computational operations, enhancing the overall efficiency of the privacy-preserving diffusion process. We successfully implement HE-based privacy-preserving stable diffusion inference. The experimental results show that HE-Diffusion achieves 500 times speedup compared with the baseline method, and reduces time cost of the homomorphically encrypted inference to the minute level. Both the performance and accuracy of the HE-Diffusion are on par with the plaintext counterpart. Our approach marks a significant step towards integrating advanced cryptographic techniques with state-of-the-art generative models, paving the way for privacy-preserving and efficient image generation in critical applications.

翻译：本文提出了一种基于同态加密的隐私保护稳定扩散框架HE-Diffusion，该框架主要聚焦于保护扩散过程的去噪阶段。HE-Diffusion是针对稳定扩散独特架构设计的定制化加密框架，能够在保障隐私与功能性的同时实现协同。为解决固有计算挑战，我们提出了一种新颖的最小失真方法，实现了高效的部分图像加密，在显著降低计算开销的同时不损害模型输出质量。此外，我们采用稀疏张量表示来加速运算过程，提升了隐私保护扩散的整体效率。我们成功实现了基于同态加密的隐私保护稳定扩散推理。实验结果表明，与基准方法相比，HE-Diffusion实现了500倍加速，并将同态加密推理的时间成本降低至分钟级。HE-Diffusion的性能与准确率均与明文版本相当。该方法标志着将先进密码学技术与尖端生成模型深度融合迈出了关键一步，为关键应用场景中的隐私保护与高效图像生成铺平了道路。