Despite the popularity of the `pre-train then fine-tune' paradigm in the NLP community, existing work quantifying energy costs and associated carbon emissions has largely focused on language model pre-training. Although a single pre-training run draws substantially more energy than fine-tuning, fine-tuning is performed more frequently by many more individual actors, and thus must be accounted for when considering the energy and carbon footprint of NLP. In order to better characterize the role of fine-tuning in the landscape of energy and carbon emissions in NLP, we perform a careful empirical study of the computational costs of fine-tuning across tasks, datasets, hardware infrastructure and measurement modalities. Our experimental results allow us to place fine-tuning energy and carbon costs into perspective with respect to pre-training and inference, and outline recommendations to NLP researchers and practitioners who wish to improve their fine-tuning energy efficiency.

翻译：尽管“预训练后微调”范式在自然语言处理领域广受欢迎，现有量化能耗及相关碳排放的研究主要集中于语言模型的预训练阶段。虽然单次预训练运行消耗的能源显著高于微调，但微调操作因其被更多独立研究者更频繁地执行，在评估自然语言处理领域的能耗与碳足迹时必须予以考量。为更准确刻画微调在自然语言处理能耗与碳排放格局中的角色，我们通过跨任务、数据集、硬件基础设施及测量模式的精细实证研究，系统分析了微调过程的计算成本。实验结果使我们能够将微调的能耗与碳排放置于预训练和推理的参照系中进行评估，并为希望提升微调能效的自然语言处理研究者与实践者提出具体建议。