Aphasia, a language disorder resulting from brain damage, requires accurate identification of specific aphasia types, such as Broca's and Wernicke's aphasia, for effective treatment. However, little attention has been paid to developing methods to detect different types of aphasia. Recognizing the importance of analyzing co-speech gestures for distinguish aphasia types, we propose a multimodal graph neural network for aphasia type detection using speech and corresponding gesture patterns. By learning the correlation between the speech and gesture modalities for each aphasia type, our model can generate textual representations sensitive to gesture information, leading to accurate aphasia type detection. Extensive experiments demonstrate the superiority of our approach over existing methods, achieving state-of-the-art results (F1 84.2\%). We also show that gesture features outperform acoustic features, highlighting the significance of gesture expression in detecting aphasia types. We provide the codes for reproducibility purposes\footnote{Code: \url{https://github.com/DSAIL-SKKU/Multimodal-Aphasia-Type-Detection_EMNLP_2023}}.

翻译：失语症是一种由脑损伤引起的语言障碍，需要准确识别特定失语症类型（如布罗卡氏失语症和韦尼克氏失语症）以实现有效治疗。然而，目前针对不同类型失语症检测方法的研究较少。考虑到共语手势分析对区分失语症类型的重要性，本文提出了一种多模态图神经网络，利用语音及其对应手势模式进行失语症类型检测。通过针对每种失语症类型学习语音与手势模态之间的相关性，我们的模型能够生成对手势信息敏感的文本表示，从而实现准确的失语症类型检测。大量实验表明，我们的方法优于现有方法，取得了最先进的结果（F1分数84.2%）。我们还发现手势特征优于声学特征，凸显了手势表达在失语症类型检测中的重要性。为便于复现实验，我们提供了相关代码（代码：\url{https://github.com/DSAIL-SKKU/Multimodal-Aphasia-Type-Detection_EMNLP_2023}）。