Machine Learning (ML) architectures have been applied to several applications that involve sensitive data, where a guarantee of users' data privacy is required. Differentially Private Stochastic Gradient Descent (DPSGD) is the state-of-the-art method to train privacy-preserving models. However, DPSGD comes at a considerable accuracy loss leading to sub-optimal privacy/utility trade-offs. Towards investigating new ground for better privacy-utility trade-off, this work questions; (i) if models' hyperparameters have any inherent impact on ML models' privacy-preserving properties, and (ii) if models' hyperparameters have any impact on the privacy/utility trade-off of differentially private models. We propose a comprehensive design space exploration of different hyperparameters such as the choice of activation functions, the learning rate and the use of batch normalization. Interestingly, we found that utility can be improved by using Bounded RELU as activation functions with the same privacy-preserving characteristics. With a drop-in replacement of the activation function, we achieve new state-of-the-art accuracy on MNIST (96.02\%), FashionMnist (84.76\%), and CIFAR-10 (44.42\%) without any modification of the learning procedure fundamentals of DPSGD.

翻译：机器学习架构已应用于多个涉及敏感数据的场景，此类应用要求保障用户数据隐私。差分隐私随机梯度下降（DPSGD）是当前训练隐私保护模型的最先进方法。然而，DPSGD会显著降低模型准确率，导致隐私/效用权衡次优。为探索更优隐私-效用权衡的新方向，本研究提出以下问题：（i）模型超参数是否对机器学习模型的隐私保护特性具有内在影响？（ii）模型超参数是否影响差分隐私模型的隐私/效用权衡？我们针对激活函数选择、学习率及批归一化使用等不同超参数开展了系统性设计空间探索。有趣的是，研究发现采用Bounded RELU作为激活函数可在保持同等隐私保护特性的前提下提升模型效用。通过直接替换激活函数，我们在不改变DPSGD学习流程基本原理的情况下，在MNIST（96.02%）、FashionMnist（84.76%）和CIFAR-10（44.42%）数据集上取得了新的最先进准确率。