In this paper, we propose to use hybrid relay-intelligent reflecting surface (HR-IRS) to improve the security performance of directional modulation (DM) system. In particular, the eavesdropper in this system works in full-duplex (FD) mode and he will eavesdrop on the confidential message (CM) as well as send malicious jamming. We aim to maximize the secrecy rate (SR) by jointly optimizing the receive beamforming, transmit beamforming and phase shift matrix (PSM) of HR-IRS. Since the optimization problem is un-convex and the variables are coupled to each other, we solve this problem by iteratively optimizing these variables. The receive beamforming and transmit beamforming are obtained based on generalized Rayleigh-Ritz theorem and Dinkelbach's Transform respectively. And for PSM, two methods, called separate optimization of PSM (SO-PSM) and joint optimization of PSM (JO-PSM) are proposed. Thus, two iterative algorithms are proposed accordingly, namely maximizing SR based on SO-PSM (Max-SR-SOP) and maximizing SR based on JO-PSM (Max-SR-JOP). The former has better performance and the latter has lower complexity. The simulation results show that when HR-IRS has sufficient power budget, the proposed Max-SR-SOP and Max-SR-JOP can enable HR-IRS-aided DM network to obtain higher SR than passive IRS-aided DM network.

翻译：本文提出采用混合中继-智能超表面（HR-IRS）来提升定向调制（DM）系统的安全性能。特别地，系统中的窃听者工作在全双工（FD）模式下，其不仅会窃听机密消息（CM），还会发送恶意干扰信号。我们旨在通过联合优化接收波束赋形、发射波束赋形以及HR-IRS的相移矩阵（PSM）来最大化保密速率（SR）。由于该优化问题是非凸的且变量相互耦合，我们通过迭代优化这些变量来求解。其中，接收波束赋形与发射波束赋形分别基于广义雷利-里兹定理和丁克尔巴赫变换得到。针对PSM，我们提出了两种方法，即PSM独立优化（SO-PSM）与PSM联合优化（JO-PSM）。相应地，本文提出了两种迭代算法：基于SO-PSM最大化SR（Max-SR-SOP）和基于JO-PSM最大化SR（Max-SR-JOP）。前者具有更优的性能，后者则具有更低的复杂度。仿真结果表明，当HR-IRS拥有充足的功率预算时，所提Max-SR-SOP与Max-SR-JOP算法能使HR-IRS辅助的DM网络获得比无源IRS辅助的DM网络更高的保密速率。