The design of message passing (MP) algorithms on factor graphs is an effective manner to implement channel estimation (CE) in wireless communication systems, which performance can be further improved by exploiting prior probability models that accurately match the channel characteristics. In this work, we study the CE problem in a downlink massive multiple-input multiple-output (MIMO) orthogonal frequency division multiplexing (OFDM) system. As the prior probability, we propose the Markov chain two-state Gaussian mixture with large variance differences (TSGM-LVD) model to exploit the structured sparsity in the angle-frequency domain of the channel. Existing single and combined MP rules cannot deal with the message computation of the proposed probability model. To overcome this issue, we present a general method to derive the hybrid message passing (HMP) rule, which allows the calculation of messages described by mixed linear and non-linear functions. Accordingly, we design the HMP-TSGM-LVD algorithm under the structured turbo framework (STF). Simulation results demonstrate that the proposed algorithm converges faster and obtains better and more stable performance than its counterparts. In particular, the gain of the proposed approach is maximum (3 dB) in the high signal-to-noise ratio regime, while benchmark approaches experience oscillating behavior due to the improper prior model characterization.

翻译：在因子图上设计消息传递（MP）算法是实现无线通信系统信道估计（CE）的有效方式，通过利用精确匹配信道特征的先验概率模型可进一步提升其性能。本文研究下行链路大规模多输入多输出（MIMO）正交频分复用（OFDM）系统中的信道估计问题。作为先验概率，我们提出马尔可夫链大方差差异双态高斯混合（TSGM-LVD）模型，以利用信道角度-频率域的结构化稀疏性。现有单一及组合MP规则无法处理所提概率模型的消息计算。为解决此问题，我们提出一种通用方法推导混合消息传递（HMP）规则，该规则允许计算由混合线性与非线性函数描述的消息。据此，我们在结构化Turbo框架（STF）下设计了HMP-TSGM-LVD算法。仿真结果表明，相比其他对比算法，所提算法收敛更快，且能获得更优、更稳定的性能。尤其在信噪比高的区域内，所提方法增益最大可达3 dB，而基准方法因先验模型表征不当而产生振荡行为。