Large Language Models (LLMs) are trained on large-scale web data, which makes it difficult to grasp the contribution of each text. This poses the risk of leaking inappropriate data such as benchmarks, personal information, and copyrighted texts in the training data. Membership Inference Attacks (MIA), which determine whether a given text is included in the model's training data, have been attracting attention. Previous studies of MIAs revealed that likelihood-based classification is effective for detecting leaks in LLMs. However, the existing methods cannot be applied to some proprietary models like ChatGPT or Claude 3 because the likelihood is unavailable to the user. In this study, we propose a Sampling-based Pseudo-Likelihood (\textbf{SPL}) method for MIA (\textbf{SaMIA}) that calculates SPL using only the text generated by an LLM to detect leaks. The SaMIA treats the target text as the reference text and multiple outputs from the LLM as text samples, calculates the degree of $n$-gram match as SPL, and determines the membership of the text in the training data. Even without likelihoods, SaMIA performed on par with existing likelihood-based methods.

翻译：大型语言模型（LLMs）是在大规模网络数据上训练的，这使得理解每个文本的贡献变得困难。这带来了训练数据中泄露不适当数据的风险，例如基准测试、个人信息和受版权保护的文本。成员推断攻击（MIA）能够确定给定文本是否包含在模型的训练数据中，因此一直备受关注。先前关于MIA的研究表明，基于似然的分类对于检测LLMs中的数据泄露是有效的。然而，现有方法无法应用于某些专有模型，如ChatGPT或Claude 3，因为这些模型的似然值对用户不可用。在本研究中，我们提出了一种基于采样的伪似然方法，用于MIA（SaMIA），该方法仅利用LLM生成的文本来计算采样伪似然以检测泄露。SaMIA将目标文本作为参考文本，将LLM的多个输出作为文本样本，计算n-gram匹配度作为采样伪似然，并判断文本在训练数据中的成员资格。即使在没有似然值的情况下，SaMIA的表现也与现有的基于似然的方法相当。