The paradigm of Differentially Private SGD~(DP-SGD) can provide a theoretical guarantee for training data in both centralized and federated settings. However, the utility degradation caused by DP-SGD limits its wide application in high-stakes tasks, such as medical image diagnosis. In addition to the necessary perturbation, the convergence issue is attributed to the information loss on the gradient clipping. In this work, we propose a general framework PCDP-SGD, which aims to compress redundant gradient norms and preserve more crucial top gradient components via projection operation before gradient clipping. Additionally, we extend PCDP-SGD as a fundamental component in differential privacy federated learning~(DPFL) for mitigating the data heterogeneous challenge and achieving efficient communication. We prove that pre-projection enhances the convergence of DP-SGD by reducing the dependence of clipping error and bias to a fraction of the top gradient eigenspace, and in theory, limits cross-client variance to improve the convergence under heterogeneous federation. Experimental results demonstrate that PCDP-SGD achieves higher accuracy compared with state-of-the-art DP-SGD variants in computer vision tasks. Moreover, PCDP-SGD outperforms current federated learning frameworks when DP is guaranteed on local training sets.

翻译：差分隐私SGD（DP-SGD）范式能够在集中式和联邦场景下为训练数据提供理论保障。然而，DP-SGD导致的效用退化限制了其在医学图像诊断等高敏感性任务中的广泛应用。除必要的扰动外，收敛性问题还归因于梯度裁剪过程中的信息损失。本文提出通用框架PCDP-SGD，旨在通过裁剪操作前的投影操作压缩冗余梯度范数，并保留更关键的高秩梯度分量。此外，我们将PCDP-SGD扩展为差分隐私联邦学习（DPFL）的基础组件，以缓解数据异质性挑战并实现高效通信。我们证明预投影通过将裁剪误差与偏差对高秩梯度特征空间的依赖性降低至极小比例，从而增强DP-SGD的收敛性，并在理论上限制跨客户端方差以提升异构联邦场景下的收敛效果。实验结果表明，在计算机视觉任务中，PCDP-SGD相比现有最优DP-SGD变体取得了更高精度。此外，当本地训练集需保证差分隐私时，PCDP-SGD的表现超越了当前联邦学习框架。