We present new auditors to assess Differential Privacy (DP) of an algorithm based on output samples. Such empirical auditors are common to check for algorithmic correctness and implementation bugs. Most existing auditors are batch-based or targeted toward the traditional notion of $(\varepsilon,δ)$-DP; typically both. In this work, we shift the focus to the highly expressive privacy concept of $f$-DP, in which the entire privacy behavior is captured by a single tradeoff curve. Our auditors detect violations across the full privacy spectrum with statistical significance guarantees, which are supported by theory and simulations. Most importantly, and in contrast to prior work, our auditors do not require a user-specified sample size as an input. Rather, they adaptively determine a near-optimal number of samples needed to reach a decision, thereby avoiding the excessively large sample sizes common in many auditing studies. This reduction in sampling cost becomes especially beneficial for expensive training procedures such as DP-SGD. Our method supports both whitebox and blackbox settings and can also be executed in single-run frameworks.

翻译：本文提出一种基于输出样本评估算法差分隐私（DP）性能的新型审计方法。此类经验性审计工具常被用于检验算法正确性及发现实现缺陷。现有审计方法大多采用批处理模式，且主要针对传统的$(\varepsilon,\delta)$-DP框架；通常二者兼有。本研究将关注点转向表达力更强的$f$-DP隐私框架，该框架通过单一权衡曲线完整刻画隐私保护行为。我们提出的审计方法能够在全隐私谱系范围内检测违规行为，并具有理论支撑与仿真验证的统计显著性保证。与先前研究最为不同的是，本方法无需用户预先指定样本量作为输入参数，而是通过自适应机制确定达到决策所需的最优样本量，从而避免了多数审计研究中常见的样本量过度膨胀问题。这种采样成本的降低对于DP-SGD等昂贵训练流程尤为有利。本方法同时支持白盒与黑盒场景，并可在单次运行框架中执行。