We propose the Data Contamination Quiz, a simple and effective approach to detect data contamination in large language models (LLMs) and estimate the amount of it. Specifically, we frame data contamination detection as a series of multiple-choice questions. We devise a quiz format wherein three perturbed versions of each dataset instance are created. These changes only include word-level perturbations, replacing words with their contextual synonyms, ensuring both the semantic and sentence structure remain exactly the same as the original instance. Together with the original instance, these perturbed versions constitute the choices in the quiz. Given that the only distinguishing signal among these choices is the exact wording, an LLM, when tasked with identifying the original instance from the choices, opts for the original if it has memorized it in its pre-training phase--a trait intrinsic to LLMs. A dataset partition is then marked as contaminated if the LLM's performance on the quiz surpasses what random chance suggests. Our evaluation spans seven datasets and their respective splits (train and test/validation) on two state-of-the-art LLMs: GPT-4 and GPT-3.5. While lacking access to the pre-training data, our results suggest that our approach not only enhances the detection of data contamination but also provides an accurate estimation of its extent, even when the contamination signal is weak.

翻译：本文提出数据污染测验（Data Contamination Quiz），一种简单有效的方法，用于检测大语言模型中的数据污染并估计其污染程度。具体而言，我们将数据污染检测构建为一系列多项选择题。我们设计了一种测验格式，为每个数据集实例生成三个扰动版本。这些改动仅包含词语级扰动，通过上下文同义词替换单词，确保语义和句子结构与原始实例完全一致。这些扰动版本与原始实例共同构成测验中的选项。鉴于选项间唯一的区分信号是精确措辞，当大语言模型被要求从选项中识别原始实例时，若其在预训练阶段已记忆该实例——这一特性内嵌于大语言模型——便会选择原始版本。若模型在该测验上的表现超过随机概率预期，则将该数据集分区标记为受污染。我们在两个前沿大语言模型——GPT-4和GPT-3.5——的七个数据集及其对应划分（训练集与测试/验证集）上进行了评估。尽管无法访问预训练数据，但结果表明，我们的方法不仅增强了对数据污染的检测能力，还能准确估计其污染程度，即便在污染信号微弱时依然有效。