Mental disorders present challenges in diagnosis and treatment due to their complex and heterogeneous nature. Electroencephalogram (EEG) has shown promise as a potential biomarker for these disorders. However, existing methods for analyzing EEG signals have limitations in addressing heterogeneity and capturing complex brain activity patterns between regions. This paper proposes a novel random effects state-space model (RESSM) for analyzing large-scale multi-channel resting-state EEG signals, accounting for the heterogeneity of brain connectivities between groups and individual subjects. We incorporate multi-level random effects for temporal dynamical and spatial mapping matrices and address nonstationarity so that the brain connectivity patterns can vary over time. The model is fitted under a Bayesian hierarchical model framework coupled with a Gibbs sampler. Compared to previous mixed-effects state-space models, we directly model high-dimensional random effects matrices without structural constraints and tackle the challenge of identifiability. Through extensive simulation studies, we demonstrate that our approach yields valid estimation and inference. We apply RESSM to a multi-site clinical trial of Major Depressive Disorder (MDD). Our analysis uncovers significant differences in resting-state brain temporal dynamics among MDD patients compared to healthy individuals. In addition, we show the subject-level EEG features derived from RESSM exhibit a superior predictive value for the heterogeneous treatment effect compared to the EEG frequency band power, suggesting the potential of EEG as a valuable biomarker for MDD.

翻译：精神疾病因其复杂性和异质性，在诊断与治疗上面临巨大挑战。脑电图（EEG）作为潜在生物标志物已展现出应用前景。然而，现有EEG信号分析方法在应对异质性及捕捉脑区间复杂活动模式方面存在局限。本文提出一种新型随机效应状态空间模型（RESSM），用于分析大规模多通道静息态EEG信号，并纳入群体与个体被试间脑连接异质性的建模。通过引入时间动态与空间映射矩阵的多层随机效应，我们解决了非平稳性问题，使脑连接模式可随时间动态变化。该模型在贝叶斯分层框架下结合吉布斯采样器进行拟合。相较于传统混合效应状态空间模型，我们直接对高维随机效应矩阵建模，无需结构约束，并攻克了参数可辨识性难题。通过大量仿真实验，验证了本方法在参数估计与统计推断中的有效性。我们将RESSM应用于一项多中心重度抑郁症（MDD）临床试验，分析发现MDD患者静息态脑区时间动态特征与健康个体存在显著差异。此外，基于RESSM提取的被试级EEG特征在预测异质性治疗效果时，其预测价值优于传统EEG频带功率，这表明EEG作为MDD生物标志物具有重要潜力。