Machine Unlearning (MU) aims to remove target training data from a trained model so that the removed data no longer influences the model's behavior, fulfilling "right to be forgotten" obligations under data privacy laws. Yet, we observe that researchers in this rapidly emerging field face challenges in analyzing and understanding the behavior of different MU methods, especially in terms of three fundamental principles in MU: accuracy, efficiency, and privacy. Consequently, they often rely on aggregate metrics and ad-hoc evaluations, making it difficult to accurately assess the trade-offs between methods. To fill this gap, we introduce a visual analytics system, Unlearning Comparator, designed to facilitate the systematic evaluation of MU methods. Our system supports two important tasks in the evaluation process: model comparison and attack simulation. First, it allows the user to compare the behaviors of two models, such as a model generated by a certain method and a retrained baseline, at class-, instance-, and layer-levels to better understand the changes made after unlearning. Second, our system simulates membership inference attacks (MIAs) to evaluate the privacy of a method, where an attacker attempts to determine whether specific data samples were part of the original training set. We evaluate our system through a case study visually analyzing prominent MU methods and demonstrate that it helps the user not only understand model behaviors but also gain insights that can inform the improvement of MU methods. The source code is publicly available at https://github.com/gnueaj/Machine-Unlearning-Comparator.

翻译：机器遗忘旨在从已训练模型中移除目标训练数据，使得被移除数据不再影响模型行为，从而满足数据隐私法规中的"被遗忘权"要求。然而，我们观察到这一新兴领域的研究人员在分析和理解不同机器遗忘方法的行为时面临挑战，特别是在机器遗忘的三个基本原则——准确性、效率和隐私性方面。因此，研究者往往依赖聚合指标和临时性评估，难以准确衡量不同方法之间的权衡关系。为填补这一空白，我们提出了一个可视化分析系统"遗忘方法比较器"，旨在促进机器遗忘方法的系统性评估。我们的系统支持评估过程中的两个重要任务：模型比较和攻击模拟。首先，该系统允许用户在类别、实例和层级三个维度上比较两个模型（如通过特定方法生成的模型与重新训练的基线模型）的行为，以更好地理解遗忘操作后的变化。其次，我们的系统通过模拟成员推理攻击来评估方法的隐私性，这种攻击试图判断特定数据样本是否属于原始训练集。我们通过案例研究对系统进行评估，可视化分析了主流的机器遗忘方法，并证明该系统不仅有助于用户理解模型行为，还能获得改进机器遗忘方法的重要洞见。源代码已在 https://github.com/gnueaj/Machine-Unlearning-Comparator 公开。