Training Data Detection (TDD) is a task aimed at determining whether a specific data instance is used to train a machine learning model. In the computer security literature, TDD is also referred to as Membership Inference Attack (MIA). Given its potential to assess the risks of training data breaches, ensure copyright authentication, and verify model unlearning, TDD has garnered significant attention in recent years, leading to the development of numerous methods. Despite these advancements, there is no comprehensive benchmark to thoroughly evaluate the effectiveness of TDD methods. In this work, we introduce TDDBench, which consists of 13 datasets spanning three data modalities: image, tabular, and text. We benchmark 21 different TDD methods across four detection paradigms and evaluate their performance from five perspectives: average detection performance, best detection performance, memory consumption, and computational efficiency in both time and memory. With TDDBench, researchers can identify bottlenecks and areas for improvement in TDD algorithms, while practitioners can make informed trade-offs between effectiveness and efficiency when selecting TDD algorithms for specific use cases. Our large-scale benchmarking also reveals the generally unsatisfactory performance of TDD algorithms across different datasets. To enhance accessibility and reproducibility, we open-source TDDBench for the research community.

翻译：训练数据检测（TDD）是一项旨在判定特定数据实例是否被用于训练机器学习模型的任务。在计算机安全文献中，TDD也被称为成员推理攻击（MIA）。鉴于其在评估训练数据泄露风险、确保版权认证以及验证模型遗忘方面的潜力，TDD近年来受到广泛关注，并催生了众多方法。尽管取得了这些进展，目前仍缺乏一个全面的基准来系统评估TDD方法的有效性。在本工作中，我们提出了TDDBench，它包含涵盖图像、表格和文本三种数据模态的13个数据集。我们对来自四种检测范式的21种不同TDD方法进行了基准测试，并从五个维度评估其性能：平均检测性能、最佳检测性能、内存消耗以及时间和内存两方面的计算效率。借助TDDBench，研究人员可以识别TDD算法的瓶颈与改进方向，而实践者则能在为特定用例选择TDD算法时，在有效性与效率之间做出明智的权衡。我们的大规模基准测试还揭示了TDD算法在不同数据集上普遍不尽人意的性能表现。为提升可访问性与可复现性，我们将TDDBench开源供研究社区使用。