Large language models (LLMs) are increasingly applied in mental health support systems, where reliable recognition of high-risk states such as suicidal ideation and self-harm is safety-critical. However, existing evaluations primarily rely on aggregate performance metrics, which often obscure risk-specific failure modes and provide limited insight into model behavior in realistic, multi-turn interactions. We present MHDash, an open-source platform designed to support the development, evaluation, and auditing of AI systems for mental health applications. MHDash integrates data collection, structured annotation, multi-turn dialogue generation, and baseline evaluation into a unified pipeline. The platform supports annotations across multiple dimensions, including Concern Type, Risk Level, and Dialogue Intent, enabling fine-grained and risk-aware analysis. Our results reveal several key findings: (i) simple baselines and advanced LLM APIs exhibit comparable overall accuracy yet diverge significantly on high-risk cases; (ii) some LLMs maintain consistent ordinal severity ranking while failing absolute risk classification, whereas others achieve reasonable aggregate scores but suffer from high false negative rates on severe categories; and (iii) performance gaps are amplified in multi-turn dialogues, where risk signals emerge gradually. These observations demonstrate that conventional benchmarks are insufficient for safety-critical mental health settings. By releasing MHDash as an open platform, we aim to promote reproducible research, transparent evaluation, and safety-aligned development of AI systems for mental health support.

翻译：大型语言模型（LLM）正日益应用于心理健康支持系统，其中对自杀意念与自伤等高危状态的可靠识别具有关键安全意义。然而，现有评估主要依赖综合性能指标，往往掩盖了风险特定的失效模式，且对模型在真实多轮交互中的行为洞察有限。本文提出MHDash——一个为心理健康应用场景下AI系统的开发、评估与审计设计的开源平台。该平台将数据收集、结构化标注、多轮对话生成与基线评估整合为统一流程，支持跨多维度（包括关切类型、风险等级与对话意图）的标注体系，实现细粒度且风险感知的分析。我们的实验结果揭示了若干关键发现：（i）简单基线模型与先进LLM API在整体准确率上表现相近，但在高风险案例中呈现显著差异；（ii）部分LLM能保持一致的序数严重度排序却无法完成绝对风险分类，而另一些模型虽获得合理的综合评分，却在严重类别上出现高假阴性率；（iii）性能差距在多轮对话中被放大，其中风险信号往往逐步显现。这些现象表明传统基准测试在安全关键的心理健康场景中存在不足。通过将MHDash作为开放平台发布，我们旨在推动心理健康支持领域AI系统的可复现研究、透明化评估与安全对齐发展。