Machine unlearning is a process to remove specific data points from a trained model while maintaining the performance on the retain data, addressing privacy or legal requirements. Despite its importance, existing unlearning evaluations tend to focus on logit-based metrics under small-scale scenarios. We observe that this could lead to a false sense of security in unlearning approaches under real-world scenarios. In this paper, we conduct a comprehensive evaluation that employs representation-based evaluations of the unlearned model under large-scale scenarios to verify whether the unlearning approaches truly eliminate the targeted data from the model's representation perspective. Our analysis reveals that current state-of-the-art unlearning approaches either completely degrade the representational quality of the unlearned model or merely modify the classifier, thereby achieving superior logit-based performance while maintaining representational similarity to the original model. Furthermore, we introduce a novel unlearning evaluation scenario in which the forgetting classes exhibit semantic similarity to downstream task classes, necessitating that feature representations diverge significantly from those of the original model, thus enabling a more thorough evaluation from a representation perspective. We hope our benchmark will serve as a standardized protocol for evaluating unlearning algorithms under realistic conditions.

翻译：机器遗忘是一种从已训练模型中移除特定数据点，同时保持对保留数据性能的过程，旨在满足隐私或法律要求。尽管其重要性，现有的遗忘评估往往侧重于小规模场景下基于逻辑值的指标。我们观察到，这可能导致在现实场景中对遗忘方法产生错误的安全感。本文通过在大规模场景下采用基于表示的方法对遗忘模型进行全面评估，以验证遗忘方法是否真正从模型表示的角度消除了目标数据。我们的分析表明，当前最先进的遗忘方法要么完全降低了遗忘模型的表示质量，要么仅修改分类器，从而在保持与原始模型表示相似性的同时，实现了优异的基于逻辑值的性能。此外，我们引入了一种新颖的遗忘评估场景，其中遗忘类别与下游任务类别存在语义相似性，这要求特征表示与原始模型的表示显著偏离，从而能够从表示角度进行更彻底的评估。我们希望我们的基准能够作为在现实条件下评估遗忘算法的标准化协议。