The ubiquity of machine learning (ML) and the demand for ever-larger models bring an increase in energy consumption and environmental impact. However, little is known about the energy scaling laws in ML, and existing research focuses on training cost -- ignoring the larger cost of inference. Furthermore, tools for measuring the energy consumption of ML do not provide actionable feedback. To address these gaps, we developed Energy Consumption Optimiser (ECOpt): a hyperparameter tuner that optimises for energy efficiency and model performance. ECOpt quantifies the trade-off between these metrics as an interpretable Pareto frontier. This enables ML practitioners to make informed decisions about energy cost and environmental impact, while maximising the benefit of their models and complying with new regulations. Using ECOpt, we show that parameter and floating-point operation counts can be unreliable proxies for energy consumption, and observe that the energy efficiency of Transformer models for text generation is relatively consistent across hardware. These findings motivate measuring and publishing the energy metrics of ML models. We further show that ECOpt can have a net positive environmental impact and use it to uncover seven models for CIFAR-10 that improve upon the state of the art, when considering accuracy and energy efficiency together.

翻译：机器学习（ML）的普及以及对日益庞大模型的需求，带来了能源消耗与环境影响的显著增长。然而，目前对机器学习中的能源缩放规律知之甚少，且现有研究多集中于训练成本——忽略了更大的推理成本。此外，现有的机器学习能耗测量工具未能提供可操作的反馈。为填补这些空白，我们开发了能源消耗优化器（ECOpt）：一种针对能源效率与模型性能进行优化的超参数调优器。ECOpt 将这两项指标间的权衡量化为一个可解释的帕累托前沿。这使得机器学习从业者能够在最大化模型效益并遵守新法规的同时，就能源成本与环境影响做出明智决策。通过使用 ECOpt，我们证明了参数量与浮点运算次数作为能耗代理指标的不可靠性，并观察到用于文本生成的 Transformer 模型在不同硬件上的能源效率相对一致。这些发现激励了对机器学习模型能源指标的测量与公布。我们进一步证明，ECOpt 能够产生净正向的环境影响，并利用它发现了七个在同时考虑准确率与能源效率时，优于 CIFAR-10 数据集上现有最佳水平的模型。