We show that it is possible to privately train convex problems that give models with similar privacy-utility trade-off as one hidden-layer ReLU networks trained with differentially private stochastic gradient descent (DP-SGD). As we show, this is possible via a certain dual formulation of the ReLU minimization problem. We derive a stochastic approximation of the dual problem that leads to a strongly convex problem which allows applying, for example, the privacy amplification by iteration type of analysis for gradient-based private optimizers, and in particular allows giving accurate privacy bounds for the noisy cyclic mini-batch gradient descent with fixed disjoint mini-batches. We obtain on the MNIST and FashionMNIST problems for the noisy cyclic mini-batch gradient descent first empirical results that show similar privacy-utility-trade-offs as DP-SGD applied to a ReLU network. We outline theoretical utility bounds that illustrate the speed-ups of the private convex approximation of ReLU networks.

翻译：我们证明，通过隐私训练凸优化问题可以获得与差分隐私随机梯度下降（DP-SGD）训练的单隐藏层ReLU网络具有相近隐私-效用权衡的模型。如本文所示，这一目标可通过ReLU最小化问题的特定对偶形式实现。我们推导出对偶问题的随机逼近形式，该形式产生强凸优化问题，从而能够应用基于梯度的隐私优化器的迭代隐私放大分析框架，特别是能够为采用固定不相交小批量的噪声循环小批量梯度下降算法提供精确的隐私界。在MNIST和FashionMNIST数据集上，我们对噪声循环小批量梯度下降算法首次获得了与DP-SGD应用于ReLU网络相当的隐私-效用权衡实证结果。我们概述了理论效用界，以阐明ReLU网络隐私凸逼近方法的加速优势。