Differentially private learning is essential for training models on sensitive data, but empirical studies consistently show that it can degrade performance, introduce fairness issues like disparate impact, and reduce adversarial robustness. The theoretical underpinnings of these phenomena in modern, non-convex neural networks remain largely unexplored. This paper introduces a unified feature-centric framework to analyze the feature learning dynamics of differentially private stochastic gradient descent (DP-SGD) in two-layer ReLU convolutional neural networks. Our analysis establishes test loss bounds governed by a crucial metric: the feature-to-noise ratio (FNR). We demonstrate that the noise required for privacy leads to suboptimal feature learning, and specifically show that: 1) imbalanced FNRs across classes and subpopulations cause disparate impact; 2) even in the same class, noise has a greater negative impact on semantically long-tailed data; and 3) noise injection exacerbates vulnerability to adversarial attacks. Furthermore, our analysis reveals that the popular paradigm of public pre-training and private fine-tuning does not guarantee improvement, particularly under significant feature distribution shifts between datasets. Experiments on synthetic and real-world data corroborate our theoretical findings.

翻译：差分隐私学习对于在敏感数据上训练模型至关重要，但实证研究一致表明，它会降低性能、引发公平性问题（如差别影响），并削弱对抗鲁棒性。这些现象在现代非凸神经网络中的理论基础在很大程度上仍未被探索。本文引入了一个统一以特征为中心的分析框架，用于研究差分隐私随机梯度下降（DP-SGD）在两层ReLU卷积神经网络中的特征学习动态。我们的分析确立了由关键指标——特征噪声比（FNR）——所限制的测试损失界限。我们证明，隐私所需的噪声会导致次优特征学习，并具体表明：1）跨类别和子群体的不平衡FNR造成差别影响；2）即使在同一类别内，噪声对语义长尾数据的负面影响更大；3）噪声注入加剧了对对抗攻击的易感性。此外，我们的分析揭示，公共预训练与私有微调这一流行范例并不能保证性能提升，尤其是在数据集之间存在显著特征分布偏移的情况下。在合成数据和真实数据上的实验证实了我们的理论发现。