Fine-tuning large language models (LLMs) is often limited by the memory available on commodity GPUs. Parameter-efficient fine-tuning (PEFT) methods such as QLoRA reduce the number of trainable parameters, yet still incur high memory usage induced by the backward pass in the full model. We revisit Ladder Side Tuning (LST), a rarely explored PEFT technique that adds a lightweight side network, and show that it matches QLoRA's compute scaling slope while cutting peak memory by 50\%. Across different downstream benchmarks spanning natural language understanding, mathematical and LLM-critic tasks, LST has competitive performance with QLoRA's accuracy on average while being much more memory-efficient. This efficiency enables fine-tuning of 7B-parameter models on a single 12 GB consumer GPU with 2k-token contexts, requiring no gradient checkpointing\textemdash conditions under which QLoRA exhausts memory. Beyond memory efficiency, we also establish scaling laws showing that LST scales similarly to QLoRA. We exploit Ladder's architectural flexibility by introducing xLadder, a depth-extended variant that increases effective depth via cross-connections and shortens chain-of-thought (CoT) at fixed parameter count. Ladder is strong when memory is the bottleneck; xLadder builds on this by enabling deeper reasoning without additional memory overhead.

翻译：大型语言模型（LLM）的微调常受限于商用GPU的可用内存。参数高效微调（PEFT）方法如QLoRA减少了可训练参数数量，但仍因完整模型的反向传播过程而产生高内存占用。我们重新审视了较少被探索的PEFT技术——阶梯侧调优（LST），该方法添加了一个轻量级侧网络，并证明其在计算扩展斜率上与QLoRA相当，同时将峰值内存削减50%。在涵盖自然语言理解、数学和LLM批判任务的不同下游基准测试中，LST平均达到与QLoRA相当的精度，同时内存效率显著更高。这种效率使得在单个12GB消费级GPU上，无需梯度检查点即可对70亿参数模型进行2k令牌上下文的微调——在此条件下QLoRA会耗尽内存。除了内存效率，我们还建立了扩展定律，表明LST的扩展方式与QLoRA相似。我们利用阶梯架构的灵活性，引入了xLadder——一种通过交叉连接增加有效深度、并在固定参数数量下缩短思维链（CoT）的深度扩展变体。LST在内存受限时表现优异；xLadder在此基础上实现了更深层次的推理，且无需额外内存开销。