Noisy gradient descent and its variants are the predominant algorithms for differentially private machine learning. It is a fundamental question to quantify their privacy leakage, yet tight characterizations remain open even in the foundational setting of convex losses. This paper improves over previous analyses by establishing (and refining) the "privacy amplification by iteration" phenomenon in the unifying framework of $f$-differential privacy--which tightly captures all aspects of the privacy loss and immediately implies tighter privacy accounting in other notions of differential privacy, e.g., $(\varepsilon,\delta)$-DP and Renyi DP. Our key technical insight is the construction of shifted interpolated processes that unravel the popular shifted-divergences argument, enabling generalizations beyond divergence-based relaxations of DP. Notably, this leads to the first exact privacy analysis in the foundational setting of strongly convex optimization. Our techniques extend to many settings: convex/strongly convex, constrained/unconstrained, full/cyclic/stochastic batches, and all combinations thereof. As an immediate corollary, we recover the $f$-DP characterization of the exponential mechanism for strongly convex optimization in Gopi et al. (2022), and moreover extend this result to more general settings.

翻译：带噪声的梯度下降及其变体是差分隐私机器学习中的主流算法。量化其隐私泄露是一个基础性问题，即使在凸损失这一奠基性设定下，其严格刻画仍悬而未决。本文通过建立（并完善）$f$-差分隐私统一框架下的"迭代隐私放大"现象，改进了现有分析——该框架能精确捕捉隐私损失的所有方面，并立即导出其他差分隐私概念（如$(\varepsilon,\delta)$-DP与Rényi DP）中更严格的隐私核算。我们的关键技术见解是构建移位插值过程，以解构流行的移位散度论证，从而推广了基于散度的DP松弛方法。值得注意的是，这首次在强凸优化的基础设定中实现了精确的隐私分析。我们的技术可推广至多种场景：凸/强凸、带约束/无约束、全批量/循环/随机批量及其任意组合。作为直接推论，我们恢复了Gopi等人（2022）针对强凸优化的指数机制的$f$-DP刻画，并将该结果推广至更一般的设定。