We present differentially private (DP) algorithms for bilevel optimization, a problem class that received significant attention lately in various machine learning applications. These are the first algorithms for such problems under standard DP constraints, and are also the first to avoid Hessian computations which are prohibitive in large-scale settings. Under the well-studied setting in which the upper-level is not necessarily convex and the lower-level problem is strongly-convex, our proposed gradient-based $(ε,δ)$-DP algorithm returns a point with hypergradient norm at most $\widetilde{\mathcal{O}}\left((\sqrt{d_\mathrm{up}}/εn)^{1/2}+(\sqrt{d_\mathrm{low}}/εn)^{1/3}\right)$ where $n$ is the dataset size, and $d_\mathrm{up}/d_\mathrm{low}$ are the upper/lower level dimensions. Our analysis covers constrained and unconstrained problems alike, accounts for mini-batch gradients, and applies to both empirical and population losses. As an application, we specialize our analysis to derive a simple private rule for tuning a regularization hyperparameter.

翻译：本文提出了针对双层优化问题的差分隐私（DP）算法，该类问题近期在多种机器学习应用中受到广泛关注。这些算法是在标准DP约束下针对此类问题的首个解决方案，同时也是首个避免海森矩阵计算的方法——该计算在大规模场景中具有极高计算代价。在上层问题非凸、下层问题强凸的经典研究设定下，我们提出的基于梯度的$(ε,δ)$-DP算法返回点的超梯度范数上界为$\widetilde{\mathcal{O}}\left((\sqrt{d_\mathrm{up}}/εn)^{1/2}+(\sqrt{d_\mathrm{low}}/εn)^{1/3}\right)$，其中$n$为数据集规模，$d_\mathrm{up}/d_\mathrm{low}$分别表示上层/下层问题的维度。我们的分析同时适用于约束与非约束问题，考虑了小批量梯度计算，并适用于经验损失与总体损失。作为应用实例，我们通过特化分析推导出用于调节正则化超参数的简洁隐私规则。