A novel coexisting passive reconfigurable intelligent surface (RIS) and active decode-and-forward (DF) relay assisted non-orthogonal multiple access (NOMA) transmission framework is proposed. In particular, two communication protocols are conceived, namely Hybrid NOMA (H-NOMA) and Full NOMA (F-NOMA). Based on the proposed two protocols, both the sum rate maximization and max-min rate fairness problems are formulated for jointly optimizing the power allocation at the access point and relay as well as the passive beamforming design at the RIS. To tackle the non-convex problems, an alternating optimization (AO) based algorithm is first developed, where the transmit power and the RIS phase-shift are alternatingly optimized by leveraging the two-dimensional search and rank-relaxed difference-of-convex (DC) programming, respectively. Then, a two-layer penalty based joint optimization (JO) algorithm is developed to jointly optimize the resource allocation coefficients within each iteration. Finally, numerical results demonstrate that: i) the proposed coexisting RIS and relay assisted transmission framework is capable of achieving a significant user performance improvement than conventional schemes without RIS or relay; ii) compared with the AO algorithm, the JO algorithm requires less execution time at the cost of a slight performance loss; and iii) the H-NOMA and F-NOMA protocols are generally preferable for ensuring user rate fairness and enhancing user sum rate, respectively.

翻译：提出了一种新型的共存的被动可重构智能表面（RIS）与主动解码转发（DF）中继辅助的非正交多址接入（NOMA）传输框架。具体地，设计了两种通信协议：混合NOMA（H-NOMA）与全NOMA（F-NOMA）。基于所提出的两种协议，分别构建了以最大化总速率和最大化最小用户速率公平性为目标的问题，旨在联合优化接入点与中继的功率分配以及RIS的被动波束赋形设计。针对这些非凸问题，首先开发了一种基于交替优化（AO）的算法，其中分别利用二维搜索和秩松弛差值凸（DC）规划交替优化发射功率与RIS相位偏移。随后，开发了一种基于两层惩罚的联合优化（JO）算法，以在每次迭代中联合优化资源分配系数。最终，数值结果表明：i) 所提出的共存的RIS与中继辅助传输框架相比无RIS或中继的传统方案能够显著提升用户性能；ii) 与AO算法相比，JO算法以轻微性能损失为代价缩短了执行时间；iii) H-NOMA与F-NOMA协议分别适用于确保用户速率公平性和提升用户总速率。