Fine-tuning language models (LMs) has yielded success on diverse downstream tasks, but as LMs grow in size, backpropagation requires a prohibitively large amount of memory. Zeroth-order (ZO) methods can in principle estimate gradients using only two forward passes but are theorized to be catastrophically slow for optimizing large models. In this work, we propose a memory-efficient zerothorder optimizer (MeZO), adapting the classical ZO-SGD method to operate in-place, thereby fine-tuning LMs with the same memory footprint as inference. For example, with a single A100 80GB GPU, MeZO can train a 30-billion parameter model, whereas fine-tuning with backpropagation can train only a 2.7B LM with the same budget. We conduct comprehensive experiments across model types (masked and autoregressive LMs), model scales (up to 66B), and downstream tasks (classification, multiple-choice, and generation). Our results demonstrate that (1) MeZO significantly outperforms in-context learning and linear probing; (2) MeZO achieves comparable performance to fine-tuning with backpropagation across multiple tasks, with up to 12x memory reduction and up to 2x GPU-hour reduction in our implementation; (3) MeZO is compatible with both full-parameter and parameter-efficient tuning techniques such as LoRA and prefix tuning; (4) MeZO can effectively optimize non-differentiable objectives (e.g., maximizing accuracy or F1). We support our empirical findings with theoretical insights, highlighting how adequate pre-training and task prompts enable MeZO to fine-tune huge models, despite classical ZO analyses suggesting otherwise.

翻译：微调语言模型已在多种下游任务中取得成功，但随着语言模型规模增长，反向传播需要极其庞大的内存量。零阶方法原则上可通过仅两次前向传播估计梯度，但理论上对优化大型模型存在灾难性缓慢问题。本文提出内存高效的零阶优化器MeZO，通过改进经典ZO-SGD方法实现原地操作，从而在保持推理级内存占用的条件下微调语言模型。例如，单张A100 80GB GPU上，MeZO可训练300亿参数模型，而同等预算下基于反向传播的微调仅能训练27亿参数模型。我们针对模型类型（掩码语言模型与自回归语言模型）、模型规模（最高660亿参数）及下游任务（分类、多项选择、生成）开展全面实验。结果表明：（1）MeZO显著优于上下文学习与线性探测；（2）MeZO在多项任务中达到与反向传播微调相当的性能，实现最高12倍内存缩减及实际运算中最高2倍GPU工时减少；（3）MeZO兼容全参数与参数高效微调技术（如LoRA、前缀微调）；（4）MeZO可有效优化不可微目标函数（如最大化准确率或F1值）。我们从理论视角佐证实验发现，揭示充足的预训练与任务提示如何驱动MeZO实现巨量模型微调——尽管经典零阶分析结论相反。