We introduce a multiple target optimization framework for DP-SGD referred to as pro-active DP. In contrast to traditional DP accountants, which are used to track the expenditure of privacy budgets, the pro-active DP scheme allows one to {\it a-priori} select parameters of DP-SGD based on a fixed privacy budget (in terms of $\epsilon$ and $\delta$) in such a way to optimize the anticipated utility (test accuracy) the most. To achieve this objective, we first propose significant improvements to the moment account method, presenting a closed-form $(\epsilon,\delta)$-DP guarantee that connects all parameters in the DP-SGD setup. Generally, DP-SGD is $(\epsilon\leq 1/2,\delta=1/N)$-DP if $\sigma=\sqrt{2(\epsilon +\ln(1/\delta))/\epsilon}$ with $T$ at least $\approx 2k^2/\epsilon$ and $(2/e)^2k^2-1/2\geq \ln(N)$, where $T$ is the total number of rounds, and $K=kN$ is the total number of gradient computations where $k$ measures $K$ in number of epochs of size $N$ of the local data set. We prove that our expression is close to tight in that if $T$ is more than a constant factor $\approx 4$ smaller than the lower bound $\approx 2k^2/\epsilon$, then the $(\epsilon,\delta)$-DP guarantee is violated. Our enhanced DP theory allows us to create a utility graph and DP calculator. These tools link privacy and utility objectives and search for optimal experiment setups, efficiently taking into account both accuracy and privacy objectives, as well as implementation goals. We furnish a comprehensive implementation flow of our proactive DP, with rigorous experiments to showcase the proof-of-concept.

翻译：我们提出一种针对DP-SGD的多目标优化框架，称为主动DP（proactive DP）。与传统用于追踪隐私预算支出的DP核算器不同，主动DP方案允许根据固定隐私预算（以$\epsilon$和$\delta$表示）先验地选择DP-SGD参数，从而最大化预期的效用（测试准确率）。为实现此目标，我们首先对矩会计方法提出重要改进，给出一个连接DP-SGD设置中所有参数的闭式$(\epsilon,\delta)$-DP保证。一般而言，当$\sigma=\sqrt{2(\epsilon+\ln(1/\delta))/\epsilon}$、轮数$T$至少约为$2k^2/\epsilon$且$(2/e)^2k^2-1/2\geq\ln(N)$时，DP-SGD满足$(\epsilon\leq 1/2,\delta=1/N)$-DP，其中$T$为总轮数，$K=kN$为梯度计算总量，$k$以本地数据集规模$N$的轮次数度量$K$。我们证明该表达式是紧的：若$T$比下界$\approx 2k^2/\epsilon$小超过约4倍的常数因子，则$(\epsilon,\delta)$-DP保证将被违反。我们增强的DP理论使我们能够创建效用图谱和DP计算器。这些工具可连接隐私与效用目标，搜索最优实验配置，并高效兼顾准确率与隐私目标及实现目标。我们提供了主动DP的完整实现流程，并通过严格实验展示了概念验证。