In this paper, we propose to use hybrid relay-intelligent reflecting surface (HR-IRS) to improve the security performance of directional modulation (DM) system. In particular, the eavesdropper in this system works in full-duplex (FD) mode and he will eavesdrop on the confidential message (CM) as well as send malicious jamming. We aim to maximize the secrecy rate (SR) by jointly optimizing the receive beamforming, transmit beamforming and phase shift matrix (PSM) of HR-IRS. Since the optimization problem is un-convex and the variables are coupled to each other, we solve this problem by iteratively optimizing these variables. The receive beamforming and transmit beamforming are obtained based on generalized Rayleigh-Ritz theorem and Dinkelbach's Transform respectively. And for PSM, two methods, called separate optimization of PSM (SO-PSM) and joint optimization of PSM (JO-PSM) are proposed. Thus, two iterative algorithms are proposed accordingly, namely maximizing SR based on SO-PSM (Max-SR-SOP) and maximizing SR based on JO-PSM (Max-SR-JOP). The former has better performance and the latter has lower complexity. The simulation results show that when HR-IRS has sufficient power budget, the proposed Max-SR-SOP and Max-SR-JOP can enable HR-IRS-aided DM network to obtain higher SR than passive IRS-aided DM network.

翻译：本文提出利用混合中继-智能反射面（HR-IRS）提升定向调制（DM）系统的安全性能。特别地，该系统内的窃听者工作在全双工（FD）模式下，既窃听机密消息（CM），又发送恶意干扰信号。我们旨在通过联合优化接收波束赋形、发射波束赋形及HR-IRS的相移矩阵（PSM）来最大化保密速率（SR）。由于该优化问题非凸且变量相互耦合，我们采用迭代优化方法求解各变量。其中，接收波束赋形与发射波束赋形分别基于广义瑞利-里兹定理和丁克尔巴赫变换获得。针对相移矩阵，我们提出两种方法：相移矩阵分离优化（SO-PSM）与相移矩阵联合优化（JO-PSM）。据此，相应提出两种迭代算法——基于SO-PSM的最大化保密速率算法（Max-SR-SOP）与基于JO-PSM的最大化保密速率算法（Max-SR-JOP），前者具有更优性能，后者计算复杂度更低。仿真结果表明，当HR-IRS具备充足功率预算时，所提Max-SR-SOP与Max-SR-JOP算法可使HR-IRS辅助的DM网络获得高于无源IRS辅助DM网络的保密速率。