Modern commercial Heating, Ventilation, and Air Conditioning (HVAC) devices form a complex and interconnected thermodynamic system with the building and outside weather conditions, and current setpoint control policies are not fully optimized for minimizing energy use and carbon emission. Given a suitable training environment, a Reinforcement Learning (RL) model is able to improve upon these policies, but training such a model, especially in a way that scales to thousands of buildings, presents many real world challenges. We propose a novel simulation-based approach, where a customized simulator is used to train the agent for each building. Our open-source simulator (available online: https://github.com/google/sbsim) is lightweight and calibrated via telemetry from the building to reach a higher level of fidelity. On a two-story, 68,000 square foot building, with 127 devices, we were able to calibrate our simulator to have just over half a degree of drift from the real world over a six-hour interval. This approach is an important step toward having a real-world RL control system that can be scaled to many buildings, allowing for greater efficiency and resulting in reduced energy consumption and carbon emissions.

翻译：现代商用供暖、通风与空调（HVAC）设备与建筑及外部天气条件共同构成一个复杂且相互关联的热力学系统，而当前的设定点控制策略尚未完全优化以实现能耗和碳排放的最小化。在合适的训练环境下，强化学习（RL）模型能够改进这些策略，但训练此类模型（尤其是要扩展到数千栋建筑的方式）面临诸多现实挑战。我们提出一种新颖的基于仿真的方法，通过为每栋建筑定制仿真器来训练智能体。我们的开源仿真器（在线获取：https://github.com/google/sbsim）轻量化且通过建筑遥测数据进行校准，以达到更高的保真度。在拥有127个设备、占地68000平方英尺的两层建筑中，我们成功将仿真器校准至六小时间隔内与实际世界的偏差略高于半摄氏度。该方法朝着构建可扩展到多栋建筑的实际RL控制系统迈出了重要一步，从而实现更高效率并降低能耗与碳排放。