Membership inference (MI) attacks threaten user privacy through determining if a given data example has been used to train a target model. However, it has been increasingly recognized that the "comparing different MI attacks" methodology used in the existing works has serious limitations. Due to these limitations, we found (through the experiments in this work) that some comparison results reported in the literature are quite misleading. In this paper, we seek to develop a comprehensive benchmark for comparing different MI attacks, called MIBench, which consists not only the evaluation metrics, but also the evaluation scenarios. And we design the evaluation scenarios from four perspectives: the distance distribution of data samples in the target dataset, the distance between data samples of the target dataset, the differential distance between two datasets (i.e., the target dataset and a generated dataset with only nonmembers), and the ratio of the samples that are made no inferences by an MI attack. The evaluation metrics consist of ten typical evaluation metrics. We have identified three principles for the proposed "comparing different MI attacks" methodology, and we have designed and implemented the MIBench benchmark with 84 evaluation scenarios for each dataset. In total, we have used our benchmark to fairly and systematically compare 15 state-of-the-art MI attack algorithms across 588 evaluation scenarios, and these evaluation scenarios cover 7 widely used datasets and 7 representative types of models. All codes and evaluations of MIBench are publicly available at https://github.com/MIBench/MIBench.github.io/blob/main/README.md.

翻译：成员推断（MI）攻击通过判定给定数据样本是否用于训练目标模型，威胁用户隐私。然而，现有研究中"比较不同MI攻击"的方法日益暴露出严重局限性。受限于这些局限性，我们通过本文实验发现，文献中报告的部分比较结果具有相当程度的误导性。本文旨在构建一个用于比较不同MI攻击的综合基准——MIBench，该基准不仅包含评估指标，还包含评估场景。我们从四个维度设计评估场景：目标数据集中样本的距离分布、目标数据集中样本间的距离、两个数据集（即目标数据集与仅含非成员数据的生成数据集）间的微分距离，以及MI攻击无法推断的样本比例。评估指标涵盖十个典型度量标准。我们为提出的"比较不同MI攻击"方法确立了三条原则，并为每个数据集设计实现了包含84个评估场景的MIBench基准。最终，我们利用该基准在588个评估场景中对15种先进MI攻击算法进行了公平系统的比较，这些场景覆盖7个广泛使用的数据集和7类代表性模型。MIBench的全部代码及评估结果于https://github.com/MIBench/MIBench.github.io/blob/main/README.md 公开。