The arguably most widely employed algorithm to train deep neural networks with Differential Privacy is DPSGD, which requires clipping and noising of per-sample gradients. This introduces a reduction in model utility compared to non-private training. Empirically, it can be observed that this accuracy degradation is strongly dependent on the model architecture. We investigated this phenomenon and, by combining components which exhibit good individual performance, distilled a new model architecture termed SmoothNet, which is characterised by increased robustness to the challenges of DP-SGD training. Experimentally, we benchmark SmoothNet against standard architectures on two benchmark datasets and observe that our architecture outperforms others, reaching an accuracy of 73.5\% on CIFAR-10 at $\varepsilon=7.0$ and 69.2\% at $\varepsilon=7.0$ on ImageNette, a state-of-the-art result compared to prior architectural modifications for DP.

翻译：培训具有差异隐私的深神经网络的所谓最广泛使用的算法是DPSGD,它需要剪裁和点播每个样本梯度,这与非私人培训相比,减少了模型效用。可以观察到,这种精确度的退化在很大程度上取决于模型结构。我们研究了这一现象,并且通过将个人表现良好的组成部分结合起来,提炼了一个称为SlumNet的新模型结构,其特点是对DP-SGD培训的挑战更加有力。实验性地,我们用两个基准数据集作为光网基准,并观察到我们的建筑结构优于其他结构,在CIFAR-10上精确度为73.5瓦雷普西隆=7.00美元,在图像网上精确度为69.2 美元,与DP以前的建筑改造相比,这是一个最先进的结果。