New regulations and increased awareness of data privacy have led to the deployment of new and more efficient differentially private mechanisms across public institutions and industries. Ensuring the correctness of these mechanisms is therefore crucial to ensure the proper protection of data. However, since differential privacy is a property of the mechanism itself, and not of an individual output, testing whether a mechanism is differentially private is not a trivial task. While ad hoc testing techniques exist under specific assumptions, no concerted effort has been made by the research community to develop a flexible and extendable tool for testing differentially private mechanisms. This paper introduces DP-Auditorium as a step advancing research in this direction. DP-Auditorium abstracts the problem of testing differential privacy into two steps: (1) measuring the distance between distributions, and (2) finding neighboring datasets where a mechanism generates output distributions maximizing such distance. From a technical point of view, we propose three new algorithms for evaluating the distance between distributions. While these algorithms are well-established in the statistics community, we provide new estimation guarantees that exploit the fact that we are only interested in verifying whether a mechanism is differentially private, and not in obtaining an exact estimate of the distance between two distributions. DP-Auditorium is easily extensible, as demonstrated in this paper by implementing a well-known approximate differential privacy testing algorithm into our library. We provide an extensive comparison to date of multiple testers across varying sample sizes and differential privacy parameters, demonstrating that there is no single tester that dominates all others, and that a combination of different techniques is required to ensure proper testing of mechanisms.

翻译：新法规的出台及公众对数据隐私意识的增强，促使公共机构及行业部署更新、更高效的差分隐私机制。确保这些机制的正确性对于保障数据安全至关重要。然而，由于差分隐私是机制本身的属性而非单个输出的特性，测试一个机制是否满足差分隐私并非易事。尽管在特定假设下存在一些特定的测试技术，但研究界尚未协同开发出灵活可扩展的差分隐私机制测试工具。本文提出的DP-Auditorium正是朝着这一方向推进的研究成果。DP-Auditorium将差分隐私测试问题抽象为两个步骤：(1) 测量分布之间的距离；(2) 寻找使得机制输出分布之间距离最大化的相邻数据集。从技术角度而言，我们提出了三种评估分布距离的新算法。尽管这些算法在统计学界已得到充分验证，但我们提供了新的估计保证——利用我们仅需验证机制是否满足差分隐私（而非精确估计两个分布之间的距离）这一事实。DP-Auditorium具有良好的可扩展性，本文通过将其库中实现的知名近似差分隐私测试算法加以实证。我们提供了迄今为止针对不同样本量与差分隐私参数的多种测试器的全面比较，结果表明不存在单一测试器能完全主导其他测试器，需组合不同技术才能确保对机制进行有效测试。