Fine-tuning is the most effective way of adapting pre-trained large language models (LLMs) to downstream applications. With the fast growth of LLM-enabled AI applications and democratization of open-souced LLMs, fine-tuning has become possible for non-expert individuals, but intensively performed LLM fine-tuning worldwide could result in significantly high energy consumption and carbon footprint, which may bring large environmental impact. Mitigating such environmental impact towards Green AI directly correlates to reducing the FLOPs of fine-tuning, but existing techniques on efficient LLM fine-tuning can only achieve limited reduction of such FLOPs, due to their ignorance of the backpropagation cost in fine-tuning. To address this limitation, in this paper we present GreenTrainer, a new LLM fine-tuning technique that adaptively evaluates different tensors' backpropagation costs and contributions to the fine-tuned model accuracy, to minimize the fine-tuning cost by selecting the most appropriate set of tensors in training. Such selection in GreenTrainer is made based on a given objective of FLOPs reduction, which can flexibly adapt to the carbon footprint in energy supply and the need in Green AI. Experiment results over multiple open-sourced LLM models and abstractive summarization datasets show that, compared to fine-tuning the whole LLM model, GreenTrainer can save up to 64% FLOPs in fine-tuning without any noticeable model accuracy loss. Compared to the existing fine-tuning techniques such as LoRa, GreenTrainer can achieve up to 4% improvement on model accuracy with on-par FLOPs reduction.

翻译：微调是将预训练大语言模型（LLM）适配至下游应用的最有效方式。随着LLM驱动的AI应用快速发展及开源LLM的民主化普及，非专业个体也能进行微调操作，但全球范围内密集进行的LLM微调可能导致极高的能耗与碳足迹，对环境产生重大影响。减轻这类环境影响的绿色AI策略，直接关联到降低微调的计算浮点运算数（FLOPs），然而现有高效LLM微调技术仅能实现有限的FLOPs缩减，其原因在于忽视了微调过程中的反向传播成本。为突破这一局限，本文提出GreenTrainer——一种新型LLM微调技术，通过自适应评估不同张量的反向传播成本及其对微调模型准确率的贡献度，在训练过程中选取最优张量集合以最小化微调成本。GreenTrainer的选择机制基于给定的FLOPs缩减目标，可灵活适配能源供应端的碳足迹与绿色AI需求。在多个开源LLM模型与摘要生成数据集上的实验表明，与全模型微调相比，GreenTrainer在不影响模型准确率的前提下最高可节省64%的微调FLOPs；与LoRa等现有微调技术相比，在保持同等FLOPs缩减效果时，可提升模型准确率最高达4%。