Estimation of intracellular gene networks has been a critical component of single-cell transcriptomic data analysis, which can provide crucial insights into the complex interplay between genes, facilitating the discovery of the biological basis of human life at single-cell resolution. Despite notable achievements, existing methodologies often falter in their practicality, primarily due to their narrow focus on simplistic linear relationships and inadequate handling of cellular heterogeneity. To bridge these gaps, we propose a joint regularized deep neural network method incorporating Mahalanobis distance-based K-means clustering (JRDNN-KM) to estimate multiple networks for various cell subgroups simultaneously, accounting for both unknown cellular heterogeneity and zero inflation, and, more importantly, complex nonlinear relationships among genes. We introduce an innovative selection layer for network construction, along with hidden layers that include both shared and subgroup-specific neurons, to capture common patterns and subgroup-specific variations across networks. Applied to real single-cell transcriptomic data from multiple tissues and species, JRDNN-KM demonstrates higher accuracy and biological interpretability in network estimation, and more accurately identifies cell subgroups compared to current state-of-the-art methods.Building on network construction, we further find hub genes with important biological implications and modules with statistical enrichment of biological processes.

翻译：细胞内基因网络的估计一直是单细胞转录组数据分析的关键组成部分，它能够揭示基因间复杂的相互作用，从而在单细胞分辨率下促进对人类生命生物学基础的理解。尽管已有显著成果，但现有方法在实际应用中往往存在不足，主要源于其仅关注简单的线性关系以及对细胞异质性处理不足。为弥补这些缺陷，我们提出了一种结合马氏距离K均值聚类的联合正则化深度神经网络方法（JRDNN-KM），该方法能够同时估计不同细胞亚群的多个网络，同时考虑未知的细胞异质性、零膨胀现象，以及更重要的——基因间复杂的非线性关系。我们引入了一个创新的网络构建选择层，以及包含共享神经元和亚群特异性神经元的隐藏层，以捕捉网络间的共同模式和亚群特异性变异。将JRDNN-KM应用于来自多种组织和物种的真实单细胞转录组数据，结果表明其在网络估计方面比当前最先进方法具有更高的准确性和生物学可解释性，并能更精确地识别细胞亚群。基于网络构建，我们进一步发现了具有重要生物学意义的枢纽基因以及具有统计学富集生物学过程的模块。