Finetuning on task-specific datasets is a widely-embraced paradigm of harnessing the powerful capability of pretrained LLMs for various downstream tasks. Due to the popularity of LLMs finetuning and its accompanying privacy concerns, differentially private (DP) finetuning of pretrained LLMs has garnered increasing attention to safeguarding the privacy of task-specific datasets. Lying at the design core of DP LLM finetuning methods is the satisfactory tradeoff between privacy, utility, and scalability. Most existing methods build upon the seminal work of DP-SGD. Despite pushing the scalability of DP-SGD to its limit, DP-SGD-based finetuning methods are unfortunately limited by the inherent inefficiency of SGD. In this paper, we investigate the potential of DP zeroth-order methods for LLM pretraining, which avoids the scalability bottleneck of SGD by approximating the gradient with the more efficient zeroth-order gradient. Rather than treating the zeroth-order method as a drop-in replacement for SGD, this paper presents a comprehensive study both theoretically and empirically. First, we propose the stagewise DP zeroth-order method that dynamically schedules key hyperparameters. This design is grounded on the synergy between DP random perturbation and the gradient approximation error of the zeroth-order method, and its effect on finetuning trajectory. Second, we further enhance the scalability by reducing the trainable parameters that are identified by repurposing a data-free pruning technique requiring no additional data or extra privacy budget. We provide theoretical analysis for both proposed methods. We conduct extensive empirical analysis on both encoder-only masked language model and decoder-only autoregressive language model, achieving impressive results in terms of scalability and utility.

翻译：基于任务特定数据集的微调是一种广泛采用的方法，旨在利用预训练大语言模型（LLM）的强大能力处理各类下游任务。由于LLM微调的普及及其伴随的隐私问题，对预训练LLM进行差分隐私（DP）微调日益受到关注，以确保任务特定数据集的隐私安全。DP LLM微调方法的设计核心在于实现隐私性、实用性与可扩展性之间的满意权衡。现有方法大多基于DP-SGD这一开创性工作。尽管DP-SGD的可扩展性已接近极限，但基于DP-SGD的微调方法受限于SGD固有的低效率问题。本文探究了DP零阶方法在LLM预训练中的潜力，该方法通过更高效的零阶梯度近似替代传统梯度，从而规避SGD的可扩展性瓶颈。本文并非简单地将零阶方法作为SGD的替代方案，而是从理论与实证两个层面展开全面研究。首先，我们提出分阶段DP零阶方法，可动态调度关键超参数。该方法的设计基于DP随机扰动与零阶方法梯度近似误差之间的协同效应，及其对微调轨迹的影响。其次，我们通过减少可训练参数进一步扩展可扩展性——这些参数通过重新利用一种无需额外数据或隐私预算的无数据剪枝技术识别。我们为两种方法提供了理论分析。通过在仅编码器掩码语言模型和仅解码器自回归语言模型上开展广泛实证分析，我们在可扩展性与实用性方面均取得了显著成果。