The Exponential Mechanism (ExpM), a differentially private optimization method, promises many advantages over Differentially Private Stochastic Gradient Descent (DPSGD), the state-of-the-art (SOTA) and de facto method for differentially private machine learning (ML). Yet, ExpM has been historically stymied from differentially private training of modern ML algorithms by two obstructions: ExpM requires a sensitivity bound for the given loss function; ExpM requires sampling from a historically intractable density. We prove a sensitivity bound for $\ell(2)$ loss, and investigate using Normalizing Flows (NFs), deep networks furnishing approximate sampling from the otherwise intractable ExpM distribution. We prove that as the NF output converges to ExpM distribution, the privacy ($\varepsilon$) of an NF sample converges to that of the ExpM distribution. Under the assumption that the NF output distribution is the ExpM distribution, we empirically test ExpM+NF against DPSGD using the SOTA implementation (Opacus \cite{opacus} with PRV accounting) in multiple classification tasks on the Adult Dataset (census data) and MIMIC-III Dataset (healthcare records) using Logistic Regression and GRU-D, a deep learning recurrent neural network with \smallsim 20K-100K parameters. In all experiments we find ExpM+NF achieves greater than 94\% of the non-private training accuracy (AUC) with $\varepsilon$-DP for $\varepsilon$ a low as $1\mathrm{e}{-3}$ -- three orders of magnitude stronger privacy with similar accuracy. Further, performance results show ExpM+NF training time is comparable to (slightly less) than DPSGD. Limitations and future directions are provided; notably, research on NF approximation accuracy and its effect on privacy are a promising avenue to substantially advancing the field. Code for these experiments \hl{will be provided after review}.

翻译：指数机制（ExpM）作为一种差分隐私优化方法，相较于当前最先进且事实上用于差分隐私机器学习（ML）的差分隐私随机梯度下降（DPSGD），展现诸多优势。然而，历史上ExpM在训练现代ML算法时受到两个障碍的阻碍：ExpM需要给定损失函数的灵敏度界；ExpM需要从历史上难以处理的密度中进行采样。我们证明了$\ell(2)$损失的灵敏度界，并研究了利用归一化流（NFs）——即提供从原本难以处理的ExpM分布中进行近似采样的深度网络。我们证明，随着NF输出收敛于ExpM分布，NF样本的隐私度（$\varepsilon$）也收敛于ExpM分布的隐私度。在假定NF输出分布即为ExpM分布的前提下，我们使用最先进的实现（Opacus \cite{opacus}及PRV核算），在Adult数据集（人口普查数据）和MIMIC-III数据集（医疗记录）上，通过逻辑回归和GRU-D（一种具有约20K-100K参数的深度学习循环神经网络）进行多项分类任务，将ExpM+NF与DPSGD进行了实证比较。在所有实验中，我们发现ExpM+NF在$\varepsilon$-DP下，即使$\varepsilon$低至$1\mathrm{e}{-3}$——即隐私强度高出三个数量级且精度相似——仍能实现超过94%的非私有训练精度（AUC）。此外，性能结果表明ExpM+NF的训练时间与DPSGD相当（略少）。文中提供了局限性及未来方向；值得注意的是，关于NF近似精度及其对隐私影响的研究，是推动该领域取得重大进展的一条有前景的途径。这些实验的代码将在审核后提供。