Mental health disorders are increasingly prevalent worldwide, creating an urgent need for innovative tools to support early diagnosis and intervention. This study explores the potential of Large Language Models (LLMs) in multimodal mental health diagnostics, specifically for detecting depression and Post Traumatic Stress Disorder through text and audio modalities. Using the E-DAIC dataset, we compare text and audio modalities to investigate whether LLMs can perform equally well or better with audio inputs. We further examine the integration of both modalities to determine if this can enhance diagnostic accuracy, which generally results in improved performance metrics. Our analysis specifically utilizes custom-formulated metrics; Modal Superiority Score and Disagreement Resolvement Score to evaluate how combined modalities influence model performance. The Gemini 1.5 Pro model achieves the highest scores in binary depression classification when using the combined modality, with an F1 score of 0.67 and a Balanced Accuracy (BA) of 77.4%, assessed across the full dataset. These results represent an increase of 3.1% over its performance with the text modality and 2.7% over the audio modality, highlighting the effectiveness of integrating modalities to enhance diagnostic accuracy. Notably, all results are obtained in zero-shot inferring, highlighting the robustness of the models without requiring task-specific fine-tuning. To explore the impact of different configurations on model performance, we conduct binary, severity, and multiclass tasks using both zero-shot and few-shot prompts, examining the effects of prompt variations on performance. The results reveal that models such as Gemini 1.5 Pro in text and audio modalities, and GPT-4o mini in the text modality, often surpass other models in balanced accuracy and F1 scores across multiple tasks.

翻译：心理健康障碍在全球范围内日益普遍，迫切需要创新工具来支持早期诊断和干预。本研究探索了大型语言模型在多模态心理健康诊断中的潜力，特别是通过文本和音频模态检测抑郁症和创伤后应激障碍。利用E-DAIC数据集，我们比较了文本和音频模态，以探究LLMs在音频输入上是否能够表现相当或更好。我们进一步研究了两种模态的整合是否能提高诊断准确性，这通常能带来性能指标的提升。我们的分析特别采用了自定义的度量标准——模态优势分数和分歧解决分数，以评估组合模态如何影响模型性能。在使用组合模态进行二元抑郁症分类时，Gemini 1.5 Pro模型在整个数据集上取得了最高分，F1分数为0.67，平衡准确率为77.4%。这些结果相较于其仅使用文本模态的性能提高了3.1%，相较于仅使用音频模态提高了2.7%，突显了整合模态以提升诊断准确性的有效性。值得注意的是，所有结果均通过零样本推理获得，这凸显了模型无需任务特定微调的鲁棒性。为了探究不同配置对模型性能的影响，我们使用零样本和少样本提示进行了二元、严重程度和多分类任务，并考察了提示变化对性能的影响。结果显示，在文本和音频模态下的Gemini 1.5 Pro模型，以及在文本模态下的GPT-4o mini模型，在多项任务中的平衡准确率和F1分数上通常优于其他模型。