We present differentially private (DP) algorithms for bilevel optimization, a problem class that received significant attention lately in various machine learning applications. These are the first DP algorithms for this task that are able to provide any desired privacy, while also avoiding Hessian computations which are prohibitive in large-scale settings. Under the well-studied setting in which the upper-level is not necessarily convex and the lower-level problem is strongly-convex, our proposed gradient-based $(\epsilon,\delta)$-DP algorithm returns a point with hypergradient norm at most $\widetilde{\mathcal{O}}\left((\sqrt{d_\mathrm{up}}/\epsilon n)^{1/2}+(\sqrt{d_\mathrm{low}}/\epsilon n)^{1/3}\right)$ where $n$ is the dataset size, and $d_\mathrm{up}/d_\mathrm{low}$ are the upper/lower level dimensions. Our analysis covers constrained and unconstrained problems alike, accounts for mini-batch gradients, and applies to both empirical and population losses.

翻译：我们提出了用于双层优化的差分隐私（DP）算法，该问题类别近来在各种机器学习应用中受到广泛关注。这些是首个能够提供任意所需隐私保证、同时避免在大规模场景中计算代价高昂的海森矩阵的该任务DP算法。在上层问题非凸、下层问题强凸这一经过充分研究的设定下，我们提出的基于梯度的$(\epsilon,\delta)$-DP算法返回一个点，其超梯度范数至多为$\widetilde{\mathcal{O}}\left((\sqrt{d_\mathrm{up}}/\epsilon n)^{1/2}+(\sqrt{d_\mathrm{low}}/\epsilon n)^{1/3}\right)$，其中$n$为数据集大小，$d_\mathrm{up}/d_\mathrm{low}$分别为上层/下层维度。我们的分析同时涵盖约束与非约束问题，考虑了小批量梯度，并适用于经验损失与总体损失。