Despite the popularity of the `pre-train then fine-tune' paradigm in the NLP community, existing work quantifying energy costs and associated carbon emissions has largely focused on language model pre-training. Although a single pre-training run draws substantially more energy than fine-tuning, fine-tuning is performed more frequently by many more individual actors, and thus must be accounted for when considering the energy and carbon footprint of NLP. In order to better characterize the role of fine-tuning in the landscape of energy and carbon emissions in NLP, we perform a careful empirical study of the computational costs of fine-tuning across tasks, datasets, hardware infrastructure and measurement modalities. Our experimental results allow us to place fine-tuning energy and carbon costs into perspective with respect to pre-training and inference, and outline recommendations to NLP researchers and practitioners who wish to improve their fine-tuning energy efficiency.

翻译：尽管NLP社区广泛采用“预训练后微调”范式，现有量化能源成本及相关碳排放的研究主要集中于语言模型预训练。虽然单次预训练消耗的能量远高于微调，但微调由更多个体参与者更频繁地执行，因此评估NLP的能源与碳足迹时必须将其纳入考量。为更清晰地定位微调在NLP能源与碳排放格局中的作用，我们针对不同任务、数据集、硬件基础设施及测量模式，对微调的计算成本进行了细致的实证研究。实验结果表明，微调的能源与碳排放成本相对于预训练和推理具有可比性，并由此向希望提升微调能效的NLP研究人员与实践者提出建议。