With the growing popularity of electric vehicles (EVs), maintaining power grid stability has become a significant challenge. To address this issue, EV charging control strategies have been developed to manage the switch between vehicle-to-grid (V2G) and grid-to-vehicle (G2V) modes for EVs. In this context, multi-agent deep reinforcement learning (MADRL) has proven its effectiveness in EV charging control. However, existing MADRL-based approaches fail to consider the natural power flow of EV charging/discharging in the distribution network and ignore driver privacy. To deal with these problems, this paper proposes a novel approach that combines multi-EV charging/discharging with a radial distribution network (RDN) operating under optimal power flow (OPF) to distribute power flow in real time. A mathematical model is developed to describe the RDN load. The EV charging control problem is formulated as a Markov Decision Process (MDP) to find an optimal charging control strategy that balances V2G profits, RDN load, and driver anxiety. To effectively learn the optimal EV charging control strategy, a federated deep reinforcement learning algorithm named FedSAC is further proposed. Comprehensive simulation results demonstrate the effectiveness and superiority of our proposed algorithm in terms of the diversity of the charging control strategy, the power fluctuations on RDN, the convergence efficiency, and the generalization ability.

翻译：随着电动汽车（EV）的日益普及，维持电网稳定性已成为一项重大挑战。为解决此问题，人们开发了电动汽车充电控制策略，以管理电动汽车在车辆到电网（V2G）与电网到车辆（G2V）模式间的切换。在此背景下，多智能体深度强化学习（MADRL）已被证明在电动汽车充电控制中具有有效性。然而，现有基于MADRL的方法未能考虑配电网中电动汽车充放电的自然潮流分布，且忽略了驾驶员隐私。为应对这些问题，本文提出一种新颖方法，将多电动汽车充放电与运行于最优潮流（OPF）下的辐射状配电网（RDN）相结合，实现潮流的实时分布。本文建立了描述RDN负载的数学模型，并将电动汽车充电控制问题建模为马尔可夫决策过程（MDP），以寻求平衡V2G收益、RDN负载与驾驶员焦虑的最优充电控制策略。为有效学习最优电动汽车充电控制策略，本文进一步提出一种名为FedSAC的联邦深度强化学习算法。综合仿真结果证明了所提算法在充电控制策略多样性、RDN功率波动、收敛效率及泛化能力方面的有效性与优越性。