The rapid advancements in artificial intelligence (AI) have primarily focused on the process of learning from data to acquire knowledgeable learning systems. As these systems are increasingly deployed in critical areas, ensuring their privacy and alignment with human values is paramount. Recently, selective forgetting (also known as machine unlearning) has shown promise for privacy and data removal tasks, and has emerged as a transformative paradigm shift in the field of AI. It refers to the ability of a model to selectively erase the influence of previously seen data, which is especially important for compliance with modern data protection regulations and for aligning models with human values. Despite its promise, selective forgetting raises significant privacy concerns, especially when the data involved come from sensitive domains. While new unlearning-induced privacy attacks are continuously proposed, each is shown to outperform its predecessors using different experimental settings, which can lead to overly optimistic and potentially unfair assessments that may disproportionately favor one particular attack over the others. In this work, we present the first comprehensive benchmark for evaluating privacy vulnerabilities in selective forgetting. We extensively investigate privacy vulnerabilities of machine unlearning techniques and benchmark privacy leakage across a wide range of victim data, state-of-the-art unlearning privacy attacks, unlearning methods, and model architectures. We systematically evaluate and identify critical factors related to unlearning-induced privacy leakage. With our novel insights, we aim to provide a standardized tool for practitioners seeking to deploy customized unlearning applications with faithful privacy assessments.

翻译：人工智能（AI）的快速发展主要集中于从数据中学习以获取知识的学习系统过程。随着这些系统在关键领域日益广泛部署，确保其隐私性及与人类价值观的对齐至关重要。近年来，选择性遗忘（亦称机器遗忘）在隐私与数据移除任务中展现出潜力，并已成为AI领域一项变革性的范式转移。它指模型能够选择性地消除先前所见数据的影响，这对于遵守现代数据保护法规以及使模型与人类价值观保持一致尤为重要。尽管前景广阔，选择性遗忘也引发了重大的隐私担忧，尤其是在涉及敏感领域数据时。尽管新的遗忘诱导隐私攻击不断被提出，但每种攻击均在不同实验设置下被证明优于其前身，这可能导致过于乐观且可能不公平的评估，从而不成比例地偏向某一特定攻击。本工作中，我们提出了首个用于评估选择性遗忘中隐私漏洞的综合基准。我们深入研究了机器遗忘技术的隐私漏洞，并对广泛的受害数据、最先进的遗忘隐私攻击、遗忘方法及模型架构中的隐私泄露进行了基准测试。我们系统性地评估并识别了与遗忘诱导隐私泄露相关的关键因素。基于我们的新见解，我们旨在为寻求部署定制化遗忘应用并需进行可靠隐私评估的实践者提供一个标准化工具。