We study differentially private (DP) stochastic optimization (SO) with loss functions whose worst-case Lipschitz parameter over all data points may be extremely large. To date, the vast majority of work on DP SO assumes that the loss is uniformly Lipschitz continuous over data (i.e. stochastic gradients are uniformly bounded over all data points). While this assumption is convenient, it often leads to pessimistic excess risk bounds. In many practical problems, the worst-case (uniform) Lipschitz parameter of the loss over all data points may be extremely large due to outliers. In such cases, the error bounds for DP SO, which scale with the worst-case Lipschitz parameter of the loss, are vacuous. To address these limitations, this work provides near-optimal excess risk bounds that do not depend on the uniform Lipschitz parameter of the loss. Building on a recent line of work (Wang et al., 2020; Kamath et al., 2022), we assume that stochastic gradients have bounded $k$-th order moments for some $k \geq 2$. Compared with works on uniformly Lipschitz DP SO, our excess risk scales with the $k$-th moment bound instead of the uniform Lipschitz parameter of the loss, allowing for significantly faster rates in the presence of outliers and/or heavy-tailed data. For convex and strongly convex loss functions, we provide the first asymptotically optimal excess risk bounds (up to a logarithmic factor). In contrast to (Wang et al., 2020; Kamath et al., 2022), our bounds do not require the loss function to be differentiable/smooth. We also devise a linear-time algorithm for smooth losses that has excess risk that is tight in certain practical parameter regimes. Additionally, our work is the first to address non-convex non-uniformly Lipschitz loss functions satisfying the Proximal-PL inequality; this covers some practical machine learning models. Our Proximal-PL algorithm has near-optimal excess risk.

翻译：我们研究具有微分隐私（DP）的随机优化（SO）问题，其中损失函数在所有数据点上的最坏情况利普希茨参数可能极大。迄今为止，绝大多数关于DP SO的研究假设损失函数在数据上具有一致利普希茨连续性（即随机梯度在所有数据点上一致有界）。尽管这一假设便于处理，但常导致悲观的过风险界。在许多实际问题中，损失函数在所有数据点上的最坏情况（一致）利普希茨参数可能因异常值而极大。此时，与最坏情况利普希茨参数成比例的DP SO误差界会失效。为克服这些局限，本文提供了不依赖于损失函数一致利普希茨参数的近最优过风险界。基于近期一系列工作（Wang et al., 2020; Kamath et al., 2022），我们假设随机梯度对某个k≥2具有有界的k阶矩。与一致利普希茨的DP SO研究相比，我们的过风险以k阶矩界而非损失函数的一致利普希茨参数为标度，从而在存在异常值和/或重尾数据时实现显著更快的收敛速率。对于凸和强凸损失函数，我们首次给出了渐进最优的过风险界（仅相差对数因子）。与（Wang et al., 2020; Kamath et al., 2022）不同，我们的界不要求损失函数可微/光滑。我们为光滑损失设计了线性时间算法，其在某些实际参数范围内具有紧致的过风险。此外，本文首次解决了满足Proximal-PL不等式的非凸非一致利普希茨损失函数，这覆盖了部分实际机器学习模型。我们的Proximal-PL算法具有近最优过风险。