We introduce a novel method for adapting diffusion models under differential privacy (DP) constraints, enabling privacy-preserving style and content transfer without fine-tuning model weights. Traditional approaches to private adaptation, such as DP-SGD, incur significant computational and memory overhead when applied to large, complex models. In addition, when adapting to small-scale specialized datasets, DP-SGD incurs large amount of noise that significantly degrades the performance. Our approach instead leverages an embedding-based technique derived from Textual Inversion (TI) and adapted with differentially private mechanisms. We apply TI to Stable Diffusion for style adaptation using two private datasets: a collection of artworks by a single artist and pictograms from the Paris 2024 Olympics. Experimental results show that the TI-based adaptation achieves superior fidelity in style transfer, even under strong privacy guarantees.

翻译：本文提出一种在差分隐私约束下自适应扩散模型的新方法，该方法能够在无需微调模型权重的情况下实现隐私保护的风格与内容迁移。传统隐私自适应方法（如DP-SGD）应用于大型复杂模型时会产生显著的计算与内存开销。此外，当面向小规模专用数据集进行自适应时，DP-SGD会引入大量噪声，严重降低模型性能。我们的方法转而采用基于嵌入的技术——该方法源自文本反演，并通过差分隐私机制进行适应性改进。我们将文本反演技术应用于Stable Diffusion模型，使用两个私有数据集进行风格自适应：单一艺术家的作品集与2024年巴黎奥运会象形图。实验结果表明，即使在强隐私保障条件下，基于文本反演的自适应方法在风格迁移任务中仍能实现更优的保真度。