The state of the art and de facto standard for differentially private machine learning (ML) is differentially private stochastic gradient descent (DPSGD). Yet, the method is inherently wasteful. By adding noise to every gradient, it diminishes the overall privacy with every gradient step. Despite 15 years of fruitful research advancing the composition theorems, sub-sampling methods, and implementation techniques, adequate accuracy and privacy is often unattainable with current private ML methods. Meanwhile, the Exponential Mechanism (ExpM), designed for private optimization, has been historically sidelined from privately training modern ML algorithms primarily because ExpM requires sampling from a historically intractable density. Despite the recent discovery of Normalizing Flow models (NFs), expressive deep networks for approximating intractable distributions, ExpM remains in the background. Our position is that leveraging NFs to circumvent historic obstructions of ExpM is a potentially transformational solution for differentially private ML worth attention. We introduce a new training method, ExpM+NF, as a potential alternative to DPSGD, and we provide experiment with logistic regression and a modern deep learning model to test whether training via ExpM+NF is viable with "good" privacy parameters. Under the assumption that the NF output distribution is the ExpM distribution, we are able to achieve $\varepsilon$ a low as $1\mathrm{e}{-3}$ -- three orders of magnitude stronger privacy with similar accuracy. This work outlines a new avenue for advancing differentially private ML, namely discovering NF approximation guarantees. Code to be provided after review.

翻译：当前差分隐私机器学习（ML）的最先进且事实标准方法是差分隐私随机梯度下降（DPSGD）。然而，该方法存在固有缺陷：在每个梯度上添加噪声，会导致每一步梯度都降低整体隐私性。尽管过去15年的丰硕研究成果推动了组合定理、子采样方法和实现技术的进步，但现有隐私ML方法往往无法同时实现足够的精度和隐私性。与此同时，专为隐私优化设计的指数机制（ExpM）在历史上一直被边缘化，未能应用于现代ML算法的隐私训练，其主要原因是ExpM需要从历史上难以处理的密度分布中采样。尽管近期发现了归一化流模型（NFs）——一种用于近似难处理分布的强表达能力深度网络——但ExpM仍未被重视。我们的观点是：利用NFs绕过ExpM的历史障碍，是值得关注的、可能具有变革意义的差分隐私ML解决方案。我们提出了一种名为ExpM+NF的新训练方法，作为DPSGD的潜在替代方案，并通过逻辑回归和现代深度学习模型的实验，测试了ExpM+NF训练是否能在“良好”隐私参数下运行。假设NF输出分布即为ExpM分布，我们成功将$\varepsilon$降至$1\mathrm{e}{-3}$——在获得相似精度的前提下，将隐私强度提升了三个数量级。本研究为推进差分隐私ML开辟了新路径，即探索NF近似保证。代码将在评审后提供。