The growing demand for customized visual content has led to the rise of personalized text-to-image (T2I) diffusion models. Despite their remarkable potential, they pose significant privacy risk when misused for malicious purposes. In this paper, we propose a novel and efficient adversarial attack method, Concept Protection by Selective Attention Manipulation (CoPSAM) which targets only the cross-attention layers of a T2I diffusion model. For this purpose, we carefully construct an imperceptible noise to be added to clean samples to get their adversarial counterparts. This is obtained during the fine-tuning process by maximizing the discrepancy between the corresponding cross-attention maps of the user-specific token and the class-specific token, respectively. Experimental validation on a subset of CelebA-HQ face images dataset demonstrates that our approach outperforms existing methods. Besides this, our method presents two important advantages derived from the qualitative evaluation: (i) we obtain better protection results for lower noise levels than our competitors; and (ii) we protect the content from unauthorized use thereby protecting the individual's identity from potential misuse.

翻译：定制化视觉内容需求的增长推动了个性化文本到图像（T2I）扩散模型的兴起。尽管这些模型展现出巨大潜力，但若被恶意滥用将带来严重的隐私风险。本文提出了一种新颖高效的对抗攻击方法——选择性注意力操纵概念保护（CoPSAM），该方法专门针对T2I扩散模型的交叉注意力层。为实现这一目标，我们精心构建了难以察觉的噪声，将其添加到干净样本中以生成对抗样本。该噪声通过在微调过程中最大化用户特定标记与类别特定标记对应交叉注意力图之间的差异而获得。在CelebA-HQ人脸图像数据集子集上的实验验证表明，我们的方法优于现有技术。此外，定性评估揭示了本方法的两个重要优势：（i）在更低噪声水平下获得更优的保护效果；（ii）能有效防止内容被未授权使用，从而保护个人身份免遭潜在滥用。