Differentially private learning is essential for training models on sensitive data, but empirical studies consistently show that it can degrade performance, introduce fairness issues like disparate impact, and reduce adversarial robustness. The theoretical underpinnings of these phenomena in modern, non-convex neural networks remain largely unexplored. This paper introduces a unified feature-centric framework to analyze the feature learning dynamics of differentially private stochastic gradient descent (DP-SGD) in two-layer ReLU convolutional neural networks. Our analysis establishes test loss bounds governed by a crucial metric: the feature-to-noise ratio (FNR). We demonstrate that the noise required for privacy leads to suboptimal feature learning, and specifically show that: 1) imbalanced FNRs across classes and subpopulations cause disparate impact; 2) even in the same class, noise has a greater negative impact on semantically long-tailed data; and 3) noise injection exacerbates vulnerability to adversarial attacks. Furthermore, our analysis reveals that the popular paradigm of public pre-training and private fine-tuning does not guarantee improvement, particularly under significant feature distribution shifts between datasets. Experiments on synthetic and real-world data corroborate our theoretical findings.

翻译：差分隐私学习对于在敏感数据上训练模型至关重要，但实证研究一致表明，它可能降低模型性能、引入诸如差异性影响等公平性问题，并削弱对抗鲁棒性。这些现象在现代非凸神经网络中的理论基础在很大程度上仍未得到探索。本文引入了一个统一的以特征为中心的框架，用于分析差分隐私随机梯度下降在双层ReLU卷积神经网络中的特征学习动态。我们的分析建立了由关键指标——特征噪声比所控制的测试损失边界。我们证明了隐私所需的噪声会导致次优的特征学习，并具体表明：1）跨类别和子群体的不平衡特征噪声比会导致差异性影响；2）即使在相同类别内，噪声对语义长尾数据的负面影响更大；3）噪声注入加剧了对对抗攻击的脆弱性。此外，我们的分析揭示，流行的公共预训练与私有微调范式并不能保证性能提升，尤其是在数据集之间存在显著特征分布偏移的情况下。在合成数据与真实数据上的实验证实了我们的理论发现。