The increasing global prevalence of mental disorders, such as depression and PTSD, requires objective and scalable diagnostic tools. Traditional clinical assessments often face limitations in accessibility, objectivity, and consistency. This paper investigates the potential of multimodal machine learning to address these challenges, leveraging the complementary information available in text, audio, and video data. Our approach involves a comprehensive analysis of various data preprocessing techniques, including novel chunking and utterance-based formatting strategies. We systematically evaluate a range of state-of-the-art embedding models for each modality and employ Convolutional Neural Networks (CNNs) and Bidirectional LSTM Networks (BiLSTMs) for feature extraction. We explore data-level, feature-level, and decision-level fusion techniques, including a novel integration of Large Language Model (LLM) predictions. We also investigate the impact of replacing Multilayer Perceptron classifiers with Support Vector Machines. We extend our analysis to severity prediction using PHQ-8 and PCL-C scores and multi-class classification (considering co-occurring conditions). Our results demonstrate that utterance-based chunking significantly improves performance, particularly for text and audio modalities. Decision-level fusion, incorporating LLM predictions, achieves the highest accuracy, with a balanced accuracy of 94.8% for depression and 96.2% for PTSD detection. The combination of CNN-BiLSTM architectures with utterance-level chunking, coupled with the integration of external LLM, provides a powerful and nuanced approach to the detection and assessment of mental health conditions. Our findings highlight the potential of MMML for developing more accurate, accessible, and personalized mental healthcare tools.

翻译：全球范围内抑郁症和创伤后应激障碍等精神障碍的日益流行，亟需客观且可扩展的诊断工具。传统的临床评估方法在可及性、客观性和一致性方面常面临局限。本文研究了多模态机器学习在应对这些挑战方面的潜力，通过综合利用文本、音频和视频数据中的互补信息。我们的方法包括对各种数据预处理技术的全面分析，其中涉及新颖的分块处理和基于话语的格式化策略。我们系统评估了针对每种模态的一系列前沿嵌入模型，并采用卷积神经网络和双向长短期记忆网络进行特征提取。我们探索了数据级、特征级和决策级融合技术，包括一种整合大型语言模型预测结果的新方法。同时研究了用支持向量机替代多层感知器分类器的影响。我们将分析扩展至使用PHQ-8和PCL-C量表进行严重程度预测，以及考虑共病情况的多类别分类。实验结果表明，基于话语的分块处理能显著提升性能，尤其在文本和音频模态中。结合大型语言模型预测的决策级融合取得了最高准确率，在抑郁症检测中达到94.8%的平衡准确率，在创伤后应激障碍检测中达到96.2%。CNN-BiLSTM架构与话语级分块处理的结合，辅以外部大型语言模型的整合，为精神健康状况的检测与评估提供了强大而精细的方法。我们的研究结果凸显了多模态机器学习在开发更精准、可及且个性化的精神健康诊疗工具方面的巨大潜力。