In this work, we conduct a systematic study of stochastic saddle point problems (SSP) and stochastic variational inequalities (SVI) under the constraint of $(\epsilon,\delta)$-differential privacy (DP) in both Euclidean and non-Euclidean setups. We first consider Lipschitz convex-concave SSPs in the $\ell_p/\ell_q$ setup, $p,q\in[1,2]$. Here, we obtain a bound of $\tilde{O}\big(\frac{1}{\sqrt{n}} + \frac{\sqrt{d}}{n\epsilon}\big)$ on the strong SP-gap, where $n$ is the number of samples and $d$ is the dimension. This rate is nearly optimal for any $p,q\in[1,2]$. Without additional assumptions, such as smoothness or linearity requirements, prior work under DP has only obtained this rate when $p=q=2$ (i.e., only in the Euclidean setup). Further, existing algorithms have each only been shown to work for specific settings of $p$ and $q$ and under certain assumptions on the loss and the feasible set, whereas we provide a general algorithm for DP SSPs whenever $p,q\in[1,2]$. Our result is obtained via a novel analysis of the recursive regularization algorithm. In particular, we develop new tools for analyzing generalization, which may be of independent interest. Next, we turn our attention towards SVIs with a monotone, bounded and Lipschitz operator and consider $\ell_p$-setups, $p\in[1,2]$. Here, we provide the first analysis which obtains a bound on the strong VI-gap of $\tilde{O}\big(\frac{1}{\sqrt{n}} + \frac{\sqrt{d}}{n\epsilon}\big)$. For $p-1=\Omega(1)$, this rate is near optimal due to existing lower bounds. To obtain this result, we develop a modified version of recursive regularization. Our analysis builds on the techniques we develop for SSPs as well as employing additional novel components which handle difficulties arising from adapting the recursive regularization framework to SVIs.

翻译：在本工作中，我们对随机鞍点问题（SSP）与随机变分不等式（SVI）在$(\epsilon,\delta)$-差分隐私（DP）约束下，于欧几里得与非欧几里得设定中进行了系统性研究。我们首先考虑$\ell_p/\ell_q$设定（$p,q\in[1,2]$）下的Lipschitz凸-凹SSP。在此，我们得到了强SP间隙的$\tilde{O}\big(\frac{1}{\sqrt{n}} + \frac{\sqrt{d}}{n\epsilon}\big)$界，其中$n$为样本数，$d$为维度。对于任意$p,q\in[1,2]$，该速率近乎最优。在没有额外假设（如光滑性或线性要求）的情况下，先前关于DP的工作仅在$p=q=2$时（即仅在欧几里得设定中）获得此速率。此外，现有算法各自仅被证明适用于特定的$p$和$q$设定，且依赖于对损失函数和可行集的某些假设，而我们为DP SSP提供了一种适用于所有$p,q\in[1,2]$情形的通用算法。我们的结果通过对递归正则化算法的新颖分析获得。具体而言，我们开发了用于分析泛化性的新工具，这可能具有独立的研究价值。接下来，我们将注意力转向具有单调、有界且Lipschitz算子的SVI，并考虑$\ell_p$设定（$p\in[1,2]$）。在此，我们首次给出了强VI间隙的$\tilde{O}\big(\frac{1}{\sqrt{n}} + \frac{\sqrt{d}}{n\epsilon}\big)$界分析。对于$p-1=\Omega(1)$，由于现有下界的存在，该速率近乎最优。为获得此结果，我们开发了递归正则化算法的改进版本。我们的分析基于为SSP开发的技术，并采用了额外的新颖组件来处理将递归正则化框架适配至SVI时产生的困难。