Energy efficiency has become an integral aspect of modern computing infrastructure design, impacting the performance, cost, scalability, and durability of production systems. The incorporation of power actuation and sensing capabilities in CPU designs is indicative of this, enabling the deployment of system software that can actively monitor and adjust energy consumption and performance at runtime. While reinforcement learning (RL) would seem ideal for the design of such energy efficiency control systems, online training presents challenges ranging from the lack of proper models for setting up an adequate simulated environment, to perturbation (noise) and reliability issues, if training is deployed on a live system. In this paper we discuss the use of offline reinforcement learning as an alternative approach for the design of an autonomous CPU power controller, with the goal of improving the energy efficiency of parallel applications at runtime without unduly impacting their performance. Offline RL sidesteps the issues incurred by online RL training by leveraging a dataset of state transitions collected from arbitrary policies prior to training. Our methodology applies offline RL to a gray-box approach to energy efficiency, combining online application-agnostic performance data (e.g., heartbeats) and hardware performance counters to ensure that the scientific objectives are met with limited performance degradation. Evaluating our method on a variety of compute-bound and memory-bound benchmarks and controlling power on a live system through Intel's Running Average Power Limit, we demonstrate that such an offline-trained agent can substantially reduce energy consumption at a tolerable performance degradation cost.

翻译：能效已成为现代计算基础设施设计不可或缺的组成部分，深刻影响着生产系统的性能、成本、可扩展性与耐久性。CPU设计中集成功率执行与传感能力正是这一趋势的体现，使得系统软件能够在运行时主动监测并调整能耗与性能。虽然强化学习看似是设计此类能效控制系统的理想方案，但在线训练面临诸多挑战：从缺乏建立合适仿真环境的精确模型，到若在实况系统上进行训练可能引发的扰动（噪声）与可靠性问题。本文探讨采用离线强化学习作为替代方案来设计自主CPU功率控制器，其目标是在运行时提升并行应用的能效，同时不过度影响其性能。离线强化学习通过利用训练前从任意策略收集的状态转移数据集，规避了在线强化学习训练所引发的问题。我们的方法将离线强化学习应用于能效灰盒方案，结合在线应用无关性能数据（如心跳信号）与硬件性能计数器，确保在有限性能损耗下达成科学目标。通过在多种计算密集与内存密集基准测试上评估本方法，并借助英特尔的运行平均功率限制对实况系统进行功率控制，我们证明此类离线训练的智能体能够以可容忍的性能损耗为代价，显著降低能耗。