Classical differential private DP-SGD implements individual clipping with random subsampling, which forces a mini-batch SGD approach. We provide a general differential private algorithmic framework that goes beyond DP-SGD and allows any possible first order optimizers (e.g., classical SGD and momentum based SGD approaches) in combination with batch clipping, which clips an aggregate of computed gradients rather than summing clipped gradients (as is done in individual clipping). The framework also admits sampling techniques beyond random subsampling such as shuffling. Our DP analysis follows the $f$-DP approach and introduces a new proof technique based on a slightly {\em stronger} adversarial model which allows us to derive simple closed form expressions and to also analyse group privacy. In particular, for $E$ epochs work and groups of size $g$, we show a $\sqrt{g E}$ DP dependency for batch clipping with shuffling.

翻译：经典差分隐私DP-SGD通过随机子采样实现个体裁剪，这必然采用小批量SGD方法。我们提出一种通用的差分隐私算法框架，该框架超越DP-SGD，允许任意一阶优化器（例如经典SGD和基于动量的SGD方法）与批处理裁剪相结合——即裁剪计算梯度的聚合结果而非对裁剪后梯度求和（如个体裁剪所作）。该框架还支持除随机子采样外的采样技术，例如洗牌。我们的差分隐私分析基于$f$-DP方法，并通过引入一种基于略强对抗模型的新证明技术，推导出简洁的闭式表达式，同时实现对群组隐私的分析。特别地，对于$E$轮次训练和规模为$g$的群组，我们证明了洗牌与批处理裁剪结合的$\sqrt{g E}$量级DP依赖关系。