The widespread practice of fine-tuning pretrained large language models (LLMs) on domain-specific data faces two major challenges in memory and privacy. First, as the size of LLMs continue to grow, encompassing billions of parameters, the memory demands of gradient-based training methods via backpropagation become prohibitively high. Second, given the tendency of LLMs to memorize and disclose sensitive training data, the privacy of fine-tuning data must be respected. To this end, we explore the potential of zeroth-order methods in differentially private optimization for fine-tuning LLMs. Zeroth-order methods, which rely solely on forward passes, substantially reduce memory consumption during training. However, directly combining them with standard differential privacy mechanism poses dimension-dependent complexity. To bridge the gap, we introduce DPZero, a novel differentially private zeroth-order algorithm with nearly dimension-independent rates. Our theoretical analysis reveals that its complexity hinges primarily on the problem's intrinsic dimension and exhibits only a logarithmic dependence on the ambient dimension. This renders DPZero a highly practical option for real-world LLMs deployments.

翻译：微调预训练大语言模型（LLM）以适应特定领域数据的广泛应用面临两大挑战：内存和隐私。首先，随着LLM规模持续增长，包含数十亿参数，基于反向传播的梯度训练方法所需内存高得令人望而却步。其次，鉴于LLM倾向于记忆并泄露敏感训练数据，微调数据的隐私必须得到尊重。为此，我们探索了零阶方法在差分隐私优化中用于微调LLM的潜力。零阶方法仅依赖前向传播，显著降低了训练期间的内存消耗。然而，将其直接与标准差分隐私机制结合会带来维度相关的复杂性。为弥合这一差距，我们引入了DPZero，一种具有近乎维度无关速率的创新差分隐私零阶算法。我们的理论分析表明，其复杂性主要取决于问题的内在维度，而对外部维度仅呈对数依赖。这使得DPZero成为实际LLM部署中极具实用性的选择。