The hidden state threat model of differential privacy (DP) assumes that the adversary has access only to the final trained machine learning (ML) model, without seeing intermediate states during training. Current privacy analyses under this model, however, are limited to convex optimization problems, reducing their applicability to multi-layer neural networks, which are essential in modern deep learning applications. Additionally, the most successful applications of the hidden state privacy analyses in classification tasks have been for logistic regression models. We demonstrate that it is possible to privately train convex problems with privacy-utility trade-offs comparable to those of one hidden-layer ReLU networks trained with DP stochastic gradient descent (DP-SGD). We achieve this through a stochastic approximation of a dual formulation of the ReLU minimization problem which results in a strongly convex problem. This enables the use of existing hidden state privacy analyses, providing accurate privacy bounds also for the noisy cyclic mini-batch gradient descent (NoisyCGD) method with fixed disjoint mini-batches. Our experiments on benchmark classification tasks show that NoisyCGD can achieve privacy-utility trade-offs comparable to DP-SGD applied to one-hidden-layer ReLU networks. Additionally, we provide theoretical utility bounds that highlight the speed-ups gained through the convex approximation.

翻译：差分隐私（DP）的隐藏状态威胁模型假设攻击者仅能访问最终训练完成的机器学习（ML）模型，而无法观测训练过程中的中间状态。然而，当前该模型下的隐私分析仅限于凸优化问题，这限制了其在现代深度学习应用中至关重要的多层神经网络上的适用性。此外，隐藏状态隐私分析在分类任务中最成功的应用目前仅限于逻辑回归模型。我们证明，通过采用随机近似方法处理ReLU最小化问题的对偶形式，可以获得与使用DP随机梯度下降（DP-SGD）训练的单隐藏层ReLU网络相当的隐私-效用权衡。该近似将原问题转化为强凸优化问题，从而能够利用现有的隐藏状态隐私分析框架，为采用固定不相交小批量的噪声循环小批量梯度下降（NoisyCGD）方法提供精确的隐私边界。在基准分类任务上的实验表明，NoisyCGD能够实现与应用于单隐藏层ReLU网络的DP-SGD相当的隐私-效用权衡。此外，我们通过理论效用边界证明了凸近似方法带来的加速效果。