Terahertz (THz) communication with ultra-wide available spectrum is a promising technique that can achieve the stringent requirement of high data rate in the next-generation wireless networks, yet its severe propagation attenuation significantly hinders its implementation in practice. Finding beam directions for a large-scale antenna array to effectively overcome severe propagation attenuation of THz signals is a pressing need. This paper proposes a novel approach of federated deep reinforcement learning (FDRL) to swiftly perform THz-beam search for multiple base stations (BSs) coordinated by an edge server in a cellular network. All the BSs conduct deep deterministic policy gradient (DDPG)-based DRL to obtain THz beamforming policy with limited channel state information (CSI). They update their DDPG models with hidden information in order to mitigate inter-cell interference. We demonstrate that the cell network can achieve higher throughput as more THz CSI and hidden neurons of DDPG are adopted. We also show that FDRL with partial model update is able to nearly achieve the same performance of FDRL with full model update, which indicates an effective means to reduce communication load between the edge server and the BSs by partial model uploading. Moreover, the proposed FDRL outperforms conventional non-learning-based and existing non-FDRL benchmark optimization methods.

翻译：太赫兹通信凭借其超宽可用频谱，是满足下一代无线网络高数据速率严苛要求的一项有前景的技术，但其严重的传播衰减显著阻碍了实际部署。如何为大规模天线阵列高效寻找波束方向以克服太赫兹信号的严重传播衰减，已成为迫切需求。本文提出一种新颖的联邦深度强化学习方法，用于在蜂窝网络中由边缘服务器协调的多个基站快速执行太赫兹波束搜索。所有基站均采用基于深度确定性策略梯度的深度强化学习，在有限信道状态信息下获取太赫兹波束成形策略。这些基站通过隐藏信息更新其DDPG模型，以减轻小区间干扰。实验表明，随着采用更多的太赫兹CSI和DDPG隐藏神经元，蜂窝网络可实现更高的吞吐量。我们还证明，采用部分模型更新的联邦深度强化学习能够近乎达到完全模型更新方案的性能，这提供了一种通过部分模型上传来降低边缘服务器与基站间通信负载的有效手段。此外，所提出的联邦深度强化学习优于传统非学习方法和现有非联邦深度强化学习基准优化方法。