Finetuning on task-specific datasets is a widely-embraced paradigm of harnessing the powerful capability of pretrained LLMs for various downstream tasks. Due to the popularity of LLMs finetuning and its accompanying privacy concerns, differentially private (DP) finetuning of pretrained LLMs has garnered increasing attention to safeguarding the privacy of task-specific datasets. Lying at the design core of DP LLM finetuning methods is the satisfactory tradeoff between privacy, utility, and scalability. Most existing methods build upon the seminal work of DP-SGD. Despite pushing the scalability of DP-SGD to its limit, DP-SGD-based finetuning methods are unfortunately limited by the inherent inefficiency of SGD. In this paper, we investigate the potential of DP zeroth-order methods for LLM pretraining, which avoids the scalability bottleneck of SGD by approximating the gradient with the more efficient zeroth-order gradient. Rather than treating the zeroth-order method as a drop-in replacement for SGD, this paper presents a comprehensive study both theoretically and empirically. First, we propose the stagewise DP zeroth-order method that dynamically schedules key hyperparameters. This design is grounded on the synergy between DP random perturbation and the gradient approximation error of the zeroth-order method, and its effect on finetuning trajectory. Second, we further enhance the scalability by reducing the trainable parameters that are identified by repurposing a data-free pruning technique requiring no additional data or extra privacy budget. We provide theoretical analysis for both proposed methods. We conduct extensive empirical analysis on both encoder-only masked language model and decoder-only autoregressive language model, achieving impressive results in terms of scalability and utility.

翻译：针对特定任务数据集进行微调已成为利用预训练大语言模型强大能力完成各类下游任务的广泛范式。由于大模型微调的普及及其伴随的隐私问题，对预训练大语言模型进行差分隐私微调日益受到关注，以保护特定任务数据集的隐私。隐私微调方法的设计核心在于实现隐私保护、模型效用与可扩展性之间的满意权衡。现有方法大多基于开创性工作DP-SGD。尽管将DP-SGD的可扩展性推至极限，但基于DP-SGD的微调方法不幸受到SGD固有低效性的限制。本文研究了差分隐私零阶方法用于大语言模型预训练的潜力，该方法通过更高效的零阶梯度逼近梯度，从而规避了SGD的可扩展性瓶颈。本文并非将零阶方法简单视作SGD的替代品，而是从理论与实证两方面展开全面研究。首先，我们提出阶段性差分隐私零阶方法，可动态调度关键超参数。该设计基于差分隐私随机扰动与零阶方法梯度逼近误差之间的协同效应及其对微调轨迹的影响。其次，我们通过减少可训练参数进一步增强可扩展性，这些参数通过改造无需额外数据或隐私预算的无数据剪枝技术进行识别。我们为这两种方法提供了理论分析。针对仅含编码器的掩码语言模型和仅含解码器的自回归语言模型进行了大量实证分析，在可扩展性和模型效用方面取得了显著成果。