Classical differential private DP-SGD implements individual clipping with random subsampling, which forces a mini-batch SGD approach. We provide a general differential private algorithmic framework that goes beyond DP-SGD and allows any possible first order optimizers (e.g., classical SGD and momentum based SGD approaches) in combination with batch clipping, which clips an aggregate of computed gradients rather than summing clipped gradients (as is done in individual clipping). The framework also admits sampling techniques beyond random subsampling such as shuffling. Our DP analysis follows the $f$-DP approach and introduces a new proof technique which allows us to derive simple closed form expressions and to also analyse group privacy. In particular, for $E$ epochs work and groups of size $g$, we show a $\sqrt{g E}$ DP dependency for batch clipping with shuffling.

翻译：经典差分隐私DP-SGD通过随机子采样实现个体裁剪，这迫使采用小批量SGD方法。我们提出一种通用的差分隐私算法框架，该框架超越DP-SGD，允许任意一阶优化器（例如经典SGD及基于动量的SGD方法）与批次裁剪相结合——后者对计算梯度的聚合结果进行裁剪，而非对裁剪后的梯度求和（如个体裁剪所做的那样）。该框架还支持随机子采样之外的采样技术，例如混洗。我们的差分隐私分析基于$f$-DP方法，并引入一种新的证明技术，由此可推导出简单的闭式表达式，并分析组隐私。特别地，对于$E$轮次工作与大小为$g$的组，我们证明了使用混洗的批次裁剪具有$\sqrt{g E}$的差分隐私依赖关系。