Low-rank adaptation (LoRA) is an efficient strategy for adapting latent diffusion models (LDMs) on a training dataset to generate specific objects by minimizing the adaptation loss. However, adapted LDMs via LoRA are vulnerable to membership inference (MI) attacks that can judge whether a particular data point belongs to private training datasets, thus facing severe risks of privacy leakage. To defend against MI attacks, we make the first effort to propose a straightforward solution: privacy-preserving LoRA (PrivateLoRA). PrivateLoRA is formulated as a min-max optimization problem where a proxy attack model is trained by maximizing its MI gain while the LDM is adapted by minimizing the sum of the adaptation loss and the proxy attack model's MI gain. However, we empirically disclose that PrivateLoRA has the issue of unstable optimization due to the large fluctuation of the gradient scale which impedes adaptation. To mitigate this issue, we propose Stable PrivateLoRA that adapts the LDM by minimizing the ratio of the adaptation loss to the MI gain, which implicitly rescales the gradient and thus stabilizes the optimization. Our comprehensive empirical results corroborate that adapted LDMs via Stable PrivateLoRA can effectively defend against MI attacks while generating high-quality images. Our code is available at https://github.com/WilliamLUO0/StablePrivateLoRA.

翻译：低秩适配（LoRA）是一种通过在训练数据集上最小化适配损失来生成特定对象的潜在扩散模型（LDM）高效策略。然而，通过LoRA适配的LDM易受成员推断（MI）攻击，此类攻击可判定特定数据点是否属于私有训练数据集，因而面临严重的隐私泄露风险。为抵御MI攻击，我们首次提出一种直接解决方案：隐私保护LoRA（PrivateLoRA）。PrivateLoRA被形式化为一个极小极大优化问题，其中代理攻击模型通过最大化其MI增益进行训练，而LDM则通过最小化适配损失与代理攻击模型MI增益之和进行适配。然而，我们通过实验发现，PrivateLoRA存在因梯度尺度剧烈波动导致的优化不稳定问题，该问题阻碍了适配过程。为缓解此问题，我们提出稳定PrivateLoRA，通过最小化适配损失与MI增益之比来适配LDM，从而隐式地重新缩放梯度并稳定优化过程。全面的实验结果表明，通过稳定PrivateLoRA适配的LDM能有效抵御MI攻击，同时生成高质量图像。我们的代码已开源在https://github.com/WilliamLUO0/StablePrivateLoRA。