Millimeter-wave (mmWave) and terahertz (THz) massive MIMO systems often rely on predefined beamforming codebooks, which are usually suboptimal in Non-Line-of-Sight (NLoS) conditions and for hardware-limited transceivers. Reinforcement Learning (RL) enables adaptive, data-driven codebook design without explicit Channel State Information (CSI), but the robustness of such algorithms in practical conditions is underexplored. This paper introduces a robust multi-agent RL framework that learns beam codebooks directly from environmental feedback, eliminating the need for prior channel knowledge. Our method is well-suited for real-world deployments facing unpredictable propagation and hardware constraints. We conduct a comprehensive analysis of three off-policy algorithms, Deep Deterministic Policy Gradient (DDPG), Twin Delayed DDPG (TD3), and Soft Actor-Critic (SAC), evaluating their resilience to hardware impairments and feedback noise. Simulations show that SAC consistently outperforms deterministic methods, achieving superior beamforming gains and stability in NLoS scenarios, even under severe impairments. These results demonstrate the promise of RL-based codebook design for robust mmWave/THz massive MIMO systems.

翻译：毫米波（mmWave）与太赫兹（THz）大规模MIMO系统通常依赖预定义的波束赋形码本，这些码本在非视距（NLoS）条件下以及对硬件受限的收发机而言往往次优。强化学习（RL）能够在无需显式信道状态信息（CSI）的情况下实现自适应的数据驱动码本设计，然而此类算法在实际条件下的鲁棒性尚未得到充分探索。本文提出一种鲁棒的多智能体强化学习框架，该框架直接从环境反馈中学习波束码本，从而消除了对先验信道知识的需求。所提方法特别适用于面临不可预测传播环境与硬件约束的实际部署场景。我们对三种离策略算法——深度确定性策略梯度（DDPG）、双延迟DDPG（TD3）及柔性演员-评论家（SAC）——进行了全面分析，评估其对硬件损伤和反馈噪声的鲁棒性。仿真结果表明，即使在严重损伤条件下，SAC在NLoS场景中仍能持续优于确定性方法，实现卓越的波束赋形增益与稳定性。这些结果证明了基于强化学习的码本设计在构建鲁棒毫米波/太赫兹大规模MIMO系统中的巨大潜力。