The ubiquity of machine learning (ML) and the demand for ever-larger models bring an increase in energy consumption and environmental impact. However, little is known about the energy scaling laws in ML, and existing research focuses on training cost -- ignoring the larger cost of inference. Furthermore, tools for measuring the energy consumption of ML do not provide actionable feedback. To address these gaps, we developed Energy Consumption Optimiser (ECOpt): a hyperparameter tuner that optimises for energy efficiency and model performance. ECOpt quantifies the trade-off between these metrics as an interpretable Pareto frontier. This enables ML practitioners to make informed decisions about energy cost and environmental impact, while maximising the benefit of their models and complying with new regulations. Using ECOpt, we show that parameter and floating-point operation counts can be unreliable proxies for energy consumption, and observe that the energy efficiency of Transformer models for text generation is relatively consistent across hardware. These findings motivate measuring and publishing the energy metrics of ML models. We further show that ECOpt can have a net positive environmental impact and use it to uncover seven models for CIFAR-10 that improve upon the state of the art, when considering accuracy and energy efficiency together.

翻译：机器学习（ML）的普及以及对日益庞大模型的需求，导致了能耗与环境影响的增加。然而，目前对机器学习中的能耗扩展规律知之甚少，且现有研究多集中于训练成本——忽略了更大的推理成本。此外，用于测量机器学习能耗的工具未能提供可操作的反馈。为填补这些空白，我们开发了能耗优化器（ECOpt）：一种针对能效与模型性能进行优化的超参数调优器。ECOpt 将这两项指标之间的权衡量化为一个可解释的帕累托前沿。这使得机器学习从业者能够在最大化模型效益并遵守新法规的同时，就能源成本与环境影响做出明智决策。通过使用 ECOpt，我们证明参数数量和浮点运算次数可能并不可靠地代表实际能耗，并观察到用于文本生成的 Transformer 模型在不同硬件上的能效相对一致。这些发现促使我们倡导测量并公布机器学习模型的能耗指标。我们进一步表明，ECOpt 能够产生净积极的环境影响，并利用它发现了七个 CIFAR-10 模型，这些模型在综合考虑准确性与能效时，性能超越了现有技术水平。