Fine-tuning Large Language Models (LLMs) has become essential for domain adaptation, but its memory-intensive property exceeds the capabilities of most GPUs. To address this challenge and democratize LLM fine-tuning, we present SlideFormer, a novel system designed for single-GPU environments. Our innovations are: (1) A lightweight asynchronous engine that treats the GPU as a sliding window and overlaps GPU computation with CPU updates and multi-tier I/O. (2) A highly efficient heterogeneous memory management scheme significantly reduces peak memory usage. (3) Optimized Triton kernels to solve key bottlenecks and integrated advanced I/O. This collaborative design enables fine-tuning of the latest 123B+ models on a single RTX 4090, supporting up to 8x larger batch sizes and 6x larger models. In evaluations, SlideFormer achieves 1.40x to 6.27x higher throughput while roughly halving CPU/GPU memory usage compared to baselines, sustaining >95% peak performance on both NVIDIA and AMD GPUs.

翻译：大型语言模型（LLM）的微调已成为领域适应的关键环节，但其内存密集型特性超出了大多数GPU的处理能力。为应对这一挑战并推动LLM微调的普及化，我们提出了SlideFormer——一个专为单GPU环境设计的新型系统。我们的创新点在于：（1）轻量级异步引擎：将GPU视为滑动窗口，实现GPU计算与CPU更新及多层I/O操作的重叠执行。（2）高效异构内存管理方案：显著降低峰值内存使用量。（3）优化的Triton内核：解决关键性能瓶颈并集成先进I/O机制。该协同设计使得在单张RTX 4090显卡上能对最新123B+参数模型进行微调，支持高达8倍的批量大小与6倍的模型规模。评估结果表明，与基线方法相比，SlideFormer在CPU/GPU内存使用量降低约50%的同时，实现了1.40倍至6.27倍的吞吐量提升，并在NVIDIA与AMD GPU上均保持>95%的峰值性能。