Identifying training data of large-scale models is critical for copyright litigation, privacy auditing, and ensuring fair evaluation. However, existing works typically treat this task as an instance-wise identification without controlling the error rate of the identified set, which cannot provide statistically reliable evidence. In this work, we formalize training data identification as a set-level inference problem and propose Provable Training Data Identification (PTDI), a distribution-free approach that enables provable and strict false identification rate control. Specifically, our method computes conformal p-values for each data point using a set of known unseen data and then develops a novel Jackknife-corrected Beta boundary (JKBB) estimator to estimate the training-data proportion of the test set, which allows us to scale these p-values. By applying the Benjamini-Hochberg (BH) procedure to the scaled p-values, we select a subset of data points with provable and strict false identification control. Extensive experiments across various models and datasets demonstrate that PTDI achieves higher power than prior methods while strictly controlling the FIR.

翻译：识别大规模模型的训练数据对于版权诉讼、隐私审计和确保公平评估至关重要。然而，现有研究通常将此任务视为实例级别的识别，而未对识别集合的错误率进行控制，这无法提供统计上可靠的证据。在本工作中，我们将训练数据识别形式化为一个集合层面的推断问题，并提出可证明训练数据识别（PTDI），这是一种无需分布假设的方法，能够实现可证明且严格的错误识别率控制。具体而言，我们的方法利用一组已知的未见数据为每个数据点计算保形p值，然后开发了一种新颖的刀切法校正Beta边界（JKBB）估计器来估计测试集中训练数据的比例，从而使我们能够对这些p值进行缩放。通过将Benjamini-Hochberg（BH）程序应用于缩放后的p值，我们选择一个数据点子集，该子集具有可证明且严格的错误识别控制。在不同模型和数据集上的大量实验表明，PTDI在严格控制错误识别率（FIR）的同时，比现有方法实现了更高的检验功效。