Genetic association studies for brain connectivity phenotypes have gained prominence due to advances in non-invasive imaging techniques and quantitative genetics. Brain connectivity traits, characterized by network configurations and unique biological structures, present distinct challenges compared to other quantitative phenotypes. Furthermore, the presence of sample relatedness in most imaging genetics studies limits the feasibility of adopting existing network-response modeling. In this paper, we fill this gap by proposing a Bayesian network-response mixed-effect model that considers a network-variate phenotype and incorporates population structures including pedigrees and unknown sample relatedness. To accommodate the inherent topological architecture associated with the genetic contributions to the phenotype, we model the effect components via a set of effect subnetworks and impose an inter-network sparsity and intra-network shrinkage to dissect the phenotypic network configurations affected by the risk genetic variant. To facilitate uncertainty quantification of signaling components from both genotype and phenotype sides, we develop a Markov chain Monte Carlo (MCMC) algorithm for posterior inference. We evaluate the performance and robustness of our model through extensive simulations. By further applying the method to study the genetic bases for brain structural connectivity using data from the Human Connectome Project with excessive family structures, we obtain plausible and interpretable results. Beyond brain connectivity genetic studies, our proposed model also provides a general linear mixed-effect regression framework for network-variate outcomes.

翻译：脑连接表型的遗传关联研究因非侵入性成像技术和定量遗传学的进展而日益受到重视。脑连接性状以网络配置和独特生物结构为特征，与其他定量表型相比面临显著挑战。此外，大多数影像遗传学研究中样本相关性的存在限制了现有网络响应建模方法的可行性。本文通过提出一种贝叶斯网络响应混合效应模型来填补这一空白，该模型考虑网络变量表型，并纳入包括谱系和未知样本相关性在内的人群结构。为适应与遗传贡献相关的表型固有拓扑结构，我们通过一组效应子网络建模效应分量，并施加网络间稀疏性与网络内收缩性以解析受风险遗传变异影响的表型网络配置。为促进基因型和表型两侧信号成分的不确定性量化，我们开发了用于后验推断的马尔可夫链蒙特卡罗（MCMC）算法，并通过大量模拟评估模型性能与稳健性。进一步将该方法应用于人脑连接组计划中具有过度家庭结构的数据，研究脑结构连接的遗传基础，获得了合理且可解释的结果。除脑连接遗传研究外，所提模型还为网络变量结果提供了通用线性混合效应回归框架。