One of the major bottlenecks for deploying popular first-order differentially private (DP) machine learning algorithms (e.g., DP-SGD) lies in their high computation and memory cost, despite the existence of optimized implementations. Zeroth-order methods have promise in mitigating the overhead, as they leverage function evaluations to approximate the gradients, hence significantly easier to privatize. While recent works have explored zeroth-order approaches in both private and non-private settings, they still suffer from relatively low utilities compared with DP-SGD, and have only been evaluated in limited application domains. In this work, we propose to leverage public information to guide and improve gradient approximation of private zeroth-order algorithms. We explore a suite of public-data-assisted zeroth-order optimizers (PAZO) with minimal overhead. We provide theoretical analyses of the PAZO framework under an assumption of the similarity between public and private data. Empirically, we demonstrate that PAZO achieves superior privacy/utility tradeoffs across vision and text tasks in both pre-training and fine-tuning settings, outperforming the best first-order baselines (with public data) especially in highly private regimes, while offering up to $16\times$ runtime speedup.

翻译：尽管存在优化实现，部署流行的一阶差分隐私（DP）机器学习算法（如DP-SGD）的主要瓶颈之一在于其高昂的计算和内存成本。零阶方法有望缓解这一开销，因为它们利用函数评估来近似梯度，从而显著简化隐私化过程。尽管近期研究在私有和非私有场景中探索了零阶方法，但与DP-SGD相比，其效用仍然较低，且仅在有限的应用领域中得到验证。本工作中，我们提出利用公共信息来引导和改进私有零阶算法的梯度近似。我们探索了一套具有最小开销的公共数据辅助零阶优化器（PAZO）。在公共数据与私有数据相似的假设下，我们对PAZO框架进行了理论分析。实证结果表明，在预训练和微调场景下的视觉与文本任务中，PAZO实现了更优的隐私/效用权衡，尤其在高度私有化场景下超越了最佳的一阶基线方法（使用公共数据），同时提供高达$16\times$的运行加速。