Raman spectroscopy (RS) has been widely used for disease diagnosis, e.g., cardiovascular disease (CVD), owing to its efficiency and component-specific testing capabilities. A series of popular deep learning methods have recently been introduced to learn nuance features from RS for binary classifications and achieved outstanding performance than conventional machine learning methods. However, these existing deep learning methods still confront some challenges in classifying subtypes of CVD. For example, the nuance between subtypes is quite hard to capture and represent by intelligent models due to the chillingly similar shape of RS sequences. Moreover, medical history information is an essential resource for distinguishing subtypes, but they are underutilized. In light of this, we propose a multi-modality multi-scale model called M3S, which is a novel deep learning method with two core modules to address these issues. First, we convert RS data to various resolution images by the Gramian angular field (GAF) to enlarge nuance, and a two-branch structure is leveraged to get embeddings for distinction in the multi-scale feature extraction module. Second, a probability matrix and a weight matrix are used to enhance the classification capacity by combining the RS and medical history data in the multi-modality data fusion module. We perform extensive evaluations of M3S and found its outstanding performance on our in-house dataset, with accuracy, precision, recall, specificity, and F1 score of 0.9330, 0.9379, 0.9291, 0.9752, and 0.9334, respectively. These results demonstrate that the M3S has high performance and robustness compared with popular methods in diagnosing CVD subtypes.

翻译：拉曼光谱（RS）凭借其高效性和成分特异性检测能力，已被广泛应用于疾病诊断，例如心血管疾病（CVD）。近年来，一系列流行的深度学习方法被引入以从RS中学习细微特征，并在二分类任务中取得了优于传统机器学习方法的性能。然而，现有深度学习方法在CVD亚型分类中仍面临挑战。例如，由于RS序列形状极其相似，智能模型难以捕捉和表征亚型间的细微差异。此外，病史信息是区分亚型的重要资源，但尚未得到充分利用。为此，我们提出了一种名为M3S的多模态多尺度模型，这是一种包含两个核心模块的新型深度学习方法。首先，通过格拉姆角场（GAF）将RS数据转换为不同分辨率的图像以放大细微差异，并利用双分支结构在多尺度特征提取模块中获取用于区分的嵌入表示。其次，在多模态数据融合模块中，通过概率矩阵和权重矩阵结合RS与病史数据，增强分类能力。我们对M3S进行了广泛评估，发现其在我们自建数据集上表现优异：准确率0.9330、精确率0.9379、召回率0.9291、特异度0.9752、F1分数0.9334。结果表明，与主流方法相比，M3S在诊断CVD亚型时具有高鲁棒性和优越性能。