Individuals with suspected rare genetic disorders often undergo multiple clinical evaluations, imaging studies, laboratory tests and genetic tests, to find a possible answer over a prolonged period of time. Addressing this "diagnostic odyssey" thus has substantial clinical, psychosocial, and economic benefits. Many rare genetic diseases have distinctive facial features, which can be used by artificial intelligence algorithms to facilitate clinical diagnosis, in prioritizing candidate diseases to be further examined by lab tests or genetic assays, or in helping the phenotype-driven reinterpretation of genome/exome sequencing data. Existing methods using frontal facial photos were built on conventional Convolutional Neural Networks (CNNs), rely exclusively on facial images, and cannot capture non-facial phenotypic traits and demographic information essential for guiding accurate diagnoses. Here we introduce GestaltMML, a multimodal machine learning (MML) approach solely based on the Transformer architecture. It integrates facial images, demographic information (age, sex, ethnicity), and clinical notes (optionally, a list of Human Phenotype Ontology terms) to improve prediction accuracy. Furthermore, we also evaluated GestaltMML on a diverse range of datasets, including 528 diseases from the GestaltMatcher Database, several in-house datasets of Beckwith-Wiedemann syndrome (BWS, over-growth syndrome with distinct facial features), Sotos syndrome (overgrowth syndrome with overlapping features with BWS), NAA10-related neurodevelopmental syndrome, Cornelia de Lange syndrome (multiple malformation syndrome), and KBG syndrome (multiple malformation syndrome). Our results suggest that GestaltMML effectively incorporates multiple modalities of data, greatly narrowing candidate genetic diagnoses of rare diseases and may facilitate the reinterpretation of genome/exome sequencing data.

翻译：疑似罕见遗传病的患者常需经历多次临床评估、影像学检查、实验室检测及基因检测，在漫长周期中寻找可能的病因。解决这一"诊断迷航"具有显著的临床、心理社会及经济价值。许多罕见遗传病具有独特的面部特征，这些特征可被人工智能算法用于辅助临床诊断：优先排序需通过实验室或基因检测进一步验证的候选疾病，或辅助基于表型的基因组/外显子组测序数据再解读。现有基于正面面部照片的方法多采用传统卷积神经网络（CNN），仅依赖面部图像，无法捕获对精准诊断至关重要的非面部表型特征及人口学信息。本文提出GestaltMML——一种基于Transformer架构的多模态机器学习（MML）方法。该方法整合面部图像、人口学信息（年龄、性别、族裔）及临床记录（可选项：人类表型本体术语列表）以提升预测准确性。此外，我们在多样化数据集上评估了GestaltMML，包括GestaltMatcher数据库中528种疾病，以及多个内部数据集：Beckwith-Wiedemann综合征（BWS，具特征性面部的过度生长综合征）、Sotos综合征（与BWS表型重叠的过度生长综合征）、NAA10相关神经发育综合征、Cornelia de Lange综合征（多发畸形综合征）及KBG综合征（多发畸形综合征）。研究结果表明，GestaltMML能有效整合多模态数据，大幅缩小罕见病候选基因诊断范围，并可能促进基因组/外显子组测序数据的再解读。