Fine-tuning on task-specific datasets is a widely-embraced paradigm of harnessing the powerful capability of pretrained LLMs for various downstream tasks. Due to the popularity of LLMs fine-tuning and its accompanying privacy concerns, differentially private (DP) fine-tuning of pretrained LLMs has been widely used to safeguarding the privacy of task-specific datasets. Lying at the design core of DP LLM fine-tuning methods is the satisfactory tradeoff among privacy, utility, and scalability. Most existing methods build upon the seminal work of DP-SGD. Despite pushing the scalability of DP-SGD to its limit, DP-SGD-based fine-tuning methods are unfortunately limited by the inherent inefficiency of SGD. In this paper, we investigate the potential of DP zeroth-order methods for LLM pretraining, which avoids the scalability bottleneck of SGD by approximating the gradient with the more efficient zeroth-order gradient. Rather than treating the zeroth-order method as a drop-in replacement for SGD, this paper presents a comprehensive study both theoretically and empirically. First, we propose the stagewise DP zeroth-order method (DP-ZOSO) that dynamically schedules key hyperparameters. This design is grounded on the synergy between DP random perturbation and the gradient approximation error of the zeroth-order method, and its effect on fine-tuning trajectory. We provide theoretical analysis for both proposed methods. We conduct extensive empirical analysis on both encoder-only masked language model and decoder-only autoregressive language model, achieving impressive results in terms of scalability and utility (compared with DPZero, DP-ZOPO improves 4.5% on SST-5, 5.5% on MNLI with RoBERTa-Large and 9.2% on CB, 3.9% on BoolQ with OPT-2.7B when $\epsilon=4$).

翻译：在特定任务数据集上进行微调是充分利用预训练大语言模型（LLM）强大能力以应对各类下游任务的广泛采用范式。鉴于LLM微调的普及及其伴随的隐私问题，对预训练LLM进行差分隐私（DP）微调已成为保护任务数据集隐私的常见手段。DP LLM微调方法的设计核心在于隐私性、实用性与可扩展性之间的满意权衡。现有方法大多基于DP-SGD这一开创性工作。尽管已将DP-SGD的可扩展性推至极限，但基于DP-SGD的微调方法仍受限于SGD固有的低效性。本文探索了DP零阶方法在LLM预训练中的潜力，通过采用更高效的零阶梯度逼近来规避SGD的可扩展性瓶颈。不同于将零阶方法简单视为SGD的替代方案，本文从理论与实证角度展开了全面研究。首先，我们提出了分阶段DP零阶方法（DP-ZOSO），该方法能动态调度关键超参数。该设计基于DP随机扰动与零阶方法梯度逼近误差之间的协同效应及其对微调轨迹的影响。我们为两种所提方法提供了理论分析。我们在仅编码器掩码语言模型和仅解码器自回归语言模型上开展了广泛实证分析，在可扩展性与实用性方面取得了显著成果（与DPZero相比，DP-ZOPO在ϵ=4时，RoBERTa-Large模型在SST-5上提升4.5%、MNLI上提升5.5%；OPT-2.7B模型在CB上提升9.2%、BoolQ上提升3.9%）。