Membership Inference attacks (MIAs) aim to predict whether a data sample was present in the training data of a machine learning model or not, and are widely used for assessing the privacy risks of language models. Most existing attacks rely on the observation that models tend to assign higher probabilities to their training samples than non-training points. However, simple thresholding of the model score in isolation tends to lead to high false-positive rates as it does not account for the intrinsic complexity of a sample. Recent work has demonstrated that reference-based attacks which compare model scores to those obtained from a reference model trained on similar data can substantially improve the performance of MIAs. However, in order to train reference models, attacks of this kind make the strong and arguably unrealistic assumption that an adversary has access to samples closely resembling the original training data. Therefore, we investigate their performance in more realistic scenarios and find that they are highly fragile in relation to the data distribution used to train reference models. To investigate whether this fragility provides a layer of safety, we propose and evaluate neighbourhood attacks, which compare model scores for a given sample to scores of synthetically generated neighbour texts and therefore eliminate the need for access to the training data distribution. We show that, in addition to being competitive with reference-based attacks that have perfect knowledge about the training data distribution, our attack clearly outperforms existing reference-free attacks as well as reference-based attacks with imperfect knowledge, which demonstrates the need for a reevaluation of the threat model of adversarial attacks.

翻译：成员推断攻击（MIAs）旨在预测数据样本是否存在于机器学习模型的训练数据中，并被广泛用于评估语言模型的隐私风险。现有的大多数攻击基于观察结果，即模型倾向于对其训练样本赋予比非训练点更高的概率。然而，孤立地对模型分数进行简单阈值化往往会导致较高的假阳性率，因为它未考虑样本的内在复杂性。近期研究表明，与在相似数据上训练的参考模型获得的分数进行比较的基于参考的攻击，可显著提升MIAs的性能。然而，为了训练参考模型，此类攻击做出了一种强烈且可论证为不现实的假设，即攻击者能够访问与原始训练数据高度相似的样本。因此，我们在更现实的场景中研究其性能，发现它们在与参考模型训练所用的数据分布高度相关的条件下极为脆弱。为探究这种脆弱性是否提供了安全层，我们提出并评估了邻域攻击，该攻击将给定样本的模型分数与合成生成的相邻文本的分数进行比较，从而消除了访问训练数据分布的需求。我们证明，除了能与对训练数据分布具有完全知识的基于参考的攻击相媲美外，我们的攻击明显优于现有无参考攻击以及知识不完善的基于参考的攻击，这表明需重新评估对抗攻击的威胁模型。