The rapid adoption of generative AI and Large Language Models (LLMs) has spurred interest in synthetic data as a privacy-preserving alternative to sensitive real-world datasets. However, generating high-utility synthetic data often carries the risk of memorizing and regurgitating private information from the training corpus. In this work, we present a customizable empirical auditing framework designed to detect and explain such data disclosures. Our framework introduces a mechanism to distinguish between "true disclosures"-where the system directly reproduces a user's information-and "phantom disclosures''-where the system incidentally generates a user's data. By partitioning input data into training and holdout sets and applying rigorous statistical hypothesis testing, we determine if observed disclosures are consistent with strict privacy baselines, such as zero-learning or specific Differential Privacy (DP) bounds. Crucially, this approach requires no model access, no canary insertion, and no reference model training -only the synthetic output and a held-out control set. We demonstrate that this framework effectively functions as a membership inference attack, providing empirical lower bounds on privacy leakage that are tighter than prior data-based auditing methods. Our approach is model-agnostic, applies to any synthetic data generation mechanism, and requires orders of magnitude fewer computational resources than shadow-model or canary-based alternatives.

翻译：生成式AI和大语言模型（LLMs）的快速普及，促使合成数据作为敏感真实数据集的隐私保护替代方案受到关注。然而，生成高可用性的合成数据往往存在记忆并复述训练语料中隐私信息的风险。本研究提出一种可定制的实证审计框架，旨在检测并解释此类数据泄露。该框架引入了一种区分机制：将系统直接复现用户信息的"真实披露"与系统偶然生成用户数据的"幻象披露"进行甄别。通过将输入数据划分为训练集和保留集，并应用严格的统计假设检验，我们可判定观测到的披露是否符合零学习或特定差分隐私（DP）边界等严格隐私基线。关键的是，该方法无需访问模型、插入金丝雀数据或训练参考模型——仅需合成输出与保留控制集。实验证明，该框架可有效作为成员推理攻击工具，提供比先前基于数据的审计方法更紧致的隐私泄露经验下界。本方法具有模型无关性，适用于任何合成数据生成机制，且计算资源需求比影子模型或金丝雀替代方案低数个数量级。