The widespread practice of fine-tuning large language models (LLMs) on domain-specific data faces two major challenges in memory and privacy. First, as the size of LLMs continues to grow, the memory demands of gradient-based training methods via backpropagation become prohibitively high. Second, given the tendency of LLMs to memorize training data, it is important to protect potentially sensitive information in the fine-tuning data from being regurgitated. Zeroth-order methods, which rely solely on forward passes, substantially reduce memory consumption during training. However, directly combining them with standard differentially private gradient descent suffers from growing model size. To bridge this gap, we introduce DPZero, a novel private zeroth-order algorithm with nearly dimension-independent rates. The memory efficiency of DPZero is demonstrated in privately fine-tuning RoBERTa on six downstream tasks.

翻译：在大规模语言模型（LLMs）上针对特定领域数据进行微调的广泛实践面临两个主要挑战：内存和隐私。首先，随着LLMs规模的持续增长，基于反向传播的梯度训练方法对内存的需求变得过高。其次，鉴于LLMs倾向于记忆训练数据，保护微调数据中潜在的敏感信息不被泄露至关重要。仅依赖前向传播的零阶方法显著降低了训练过程中的内存消耗。然而，将其直接与标准差分隐私梯度下降结合会面临模型规模增长的问题。为弥补这一差距，我们提出DPZero，一种具有近乎与维度无关速率的全新私有零阶算法。DPZero的内存效率通过在六个下游任务上对RoBERTa进行私有微调得到验证。