Improving system-level resiliency of networked microgrids is an important aspect with increased population of inverter-based resources (IBRs). This paper (1) presents resilient control design in presence of adversarial cyber-events, and proposes a novel federated reinforcement learning (Fed-RL) approach to tackle (a) model complexities, unknown dynamical behaviors of IBR devices, (b) privacy issues regarding data sharing in multi-party-owned networked grids, and (2) transfers learned controls from simulation to hardware-in-the-loop test-bed, thereby bridging the gap between simulation and real world. With these multi-prong objectives, first, we formulate a reinforcement learning (RL) training setup generating episodic trajectories with adversaries (attack signal) injected at the primary controllers of the grid forming (GFM) inverters where RL agents (or controllers) are being trained to mitigate the injected attacks. For networked microgrids, the horizontal Fed-RL method involving distinct independent environments is not appropriate, leading us to develop vertical variant Federated Soft Actor-Critic (FedSAC) algorithm to grasp the interconnected dynamics of networked microgrid. Next, utilizing OpenAI Gym interface, we built a custom simulation set-up in GridLAB-D/HELICS co-simulation platform, named Resilient RL Co-simulation (ResRLCoSIM), to train the RL agents with IEEE 123-bus benchmark test systems comprising 3 interconnected microgrids. Finally, the learned policies in simulation world are transferred to the real-time hardware-in-the-loop test-bed set-up developed using high-fidelity Hypersim platform. Experiments show that the simulator-trained RL controllers produce convincing results with the real-time test-bed set-up, validating the minimization of sim-to-real gap.

翻译：提升网络微电网系统级弹性是伴随逆变器资源（IBR）日益普及的重要课题。本文（1）提出了对抗网络攻击事件下的弹性控制设计，并创新性地提出联邦强化学习（Fed-RL）方法，以应对（a）模型复杂度与IBR设备未知动态行为、（b）多方共有网络电网数据共享中的隐私问题；（2）将学习到的控制策略从仿真迁移至硬件在环实验平台，从而弥合仿真与现实世界的差距。基于这些多维度目标，首先构建强化学习（RL）训练框架，生成在主控制器（电网形成逆变器）中注入攻击信号的情景轨迹，RL智能体（或控制器）通过训练消除注入攻击。针对网络微电网，水平Fed-RL方法因涉及独立异构环境而不适用，因此我们开发了垂直变体联邦软演员-评论家（FedSAC）算法来捕捉网络微电网的互联动态特性。其次，利用OpenAI Gym接口，在GridLAB-D/HELICS联合仿真平台上构建了自定义仿真环境（弹性RL联合仿真平台ResRLCoSIM），基于包含三个互联微电网的IEEE 123节点基准测试系统训练RL智能体。最后，将仿真环境学习到的策略迁移至基于高保真Hypersim平台开发的实时硬件在环实验平台。实验表明，仿真训练的RL控制器在实时实验平台上产生可靠结果，验证了仿真到现实差距的最小化。