Depression is one of the most prevalent mental health disorders globally. In recent years, multi-modal data, such as speech, video, and transcripts, has been increasingly used to develop AI-assisted depression assessment systems. Large language models have further advanced this field due to their strong language understanding and generalization capabilities. However, conventional LLMs remain text-centric and cannot process the rich non-verbal cues found in audio and visual modalities, which are critical components in mental health evaluation. While multi-modal LLMs offer a promising direction, few are tailored for psychological applications. In this study, we propose a novel multi-modal LLM framework for depression detection. Our approach augments an audio language model with visual understanding and aligns audio-visual features at the timestamp level. This fine-grained alignment improves modeling of temporal dynamics across modalities while reducing the need for extensive training data and computational resources. Experiments on the DAIC-WoZ dataset demonstrate that our model outperforms both single-modality approaches and previous multi-modal methods. Moreover, the proposed framework can be extended to incorporate additional physiological signals, paving the way for broader clinical applications beyond mental health.

翻译：抑郁症是全球范围内最普遍的心理健康障碍之一。近年来，语音、视频和文本等多模态数据被越来越多地用于开发AI辅助的抑郁症评估系统。大语言模型凭借其强大的语言理解和泛化能力，进一步推动了该领域的发展。然而，传统的大语言模型仍以文本为中心，无法处理音频和视觉模态中丰富的非语言线索，而这些线索是心理健康评估的关键组成部分。尽管多模态大语言模型提供了一个有前景的方向，但鲜有针对心理学应用进行专门设计的模型。在本研究中，我们提出了一种用于抑郁症检测的新型多模态大语言模型框架。我们的方法通过视觉理解增强音频语言模型，并在时间戳级别对齐视听特征。这种细粒度对齐改善了跨模态时间动态的建模，同时减少了对大量训练数据和计算资源的需求。在DAIC-WoZ数据集上的实验表明，我们的模型在性能上优于单模态方法和先前的多模态方法。此外，所提出的框架可以扩展以整合额外的生理信号，为超越心理健康领域的更广泛临床应用铺平道路。