Recent advances in large language models have brought immense value to the world, with their superior capabilities stemming from the massive number of parameters they utilize. However, even the GPUs with the highest memory capacities, currently peaking at 80GB, are far from sufficient to accommodate these vast parameters and their associated optimizer states when conducting stochastic gradient descent-based optimization. One approach to hosting such huge models is to aggregate device memory from many GPUs. However, this approach introduces prohibitive costs for most academic researchers, who always have a limited budget for many high-end GPU servers. In this paper, we focus on huge model fine-tuning on a single, even low-end, GPU in a commodity server, which is accessible to most AI researchers. In such a scenario, the state-of-the-art work ZeRO-Infinity suffers from two severe issues when running in a commodity server: 1) low GPU utilization due to inefficient swapping, and 2) limited trainable model size due to CPU memory capacity. The underlying reason is that ZeRO-Infinity is optimized for running on high-end GPU servers. To this end, we present Fuyou, a low-cost training framework that enables efficient 100B huge model fine-tuning on a low-end server with a low-end GPU and limited CPU memory capacity. The key idea is to add the SSD-CPU communication as an optimization dimension and thus carefully co-optimize computation and data swapping from a systematic approach to maximize GPU utilization. The experimental results show that 1) Fuyou is able to fine-tune 175B GPT-3 on a consumer GPU RTX 4090 with high GPU utilization, while ZeRO-Infinity fails to fine-tune; and 2) when training a small GPT-3 13B model, Fuyou achieves 156 TFLOPS on an RTX 4090 GPU while ZeRO-Infinity only achieves 45 TFLOPS.

翻译：近年来，大语言模型的进步为世界带来了巨大价值，其卓越能力源于所使用的庞大参数量。然而，即便是当前最高内存容量达80GB的GPU，在进行基于随机梯度下降的优化时，仍远不足以容纳这些海量参数及其相关的优化器状态。一种托管此类大模型的方法是聚合多个GPU的设备内存，但该方法对预算有限的学术研究者而言成本过高，因其需要大量高端GPU服务器。本文聚焦于在普通服务器中的单个（甚至低端）GPU上实现大模型微调，这一场景对多数AI研究者具有可及性。在此场景下，当前最先进的ZeRO-Infinity框架在普通服务器运行时存在两个严重问题：1) 低效的内存交换导致GPU利用率低下；2) CPU内存容量限制训练模型尺寸。究其原因，ZeRO-Infinity优化目标为高端GPU服务器。为此，我们提出Fuyou——一种低成本的训练框架，能在配备低端GPU和有限CPU内存容量的低端服务器上高效实现100B大模型微调。核心思路是将SSD-CPU通信作为优化维度，通过系统化方法协同优化计算与数据交换，最大化GPU利用率。实验结果表明：1) Fuyou能在消费级GPU RTX 4090上以高GPU利用率微调175B参数的GPT-3模型，而ZeRO-Infinity无法完成微调；2) 在训练13B参数的GPT-3小模型时，Fuyou在RTX 4090上实现156 TFLOPS，而ZeRO-Infinity仅达到45 TFLOPS。