Current medical language model (LM) benchmarks often over-simplify the complexities of day-to-day clinical practice tasks and instead rely on evaluating LMs on multiple-choice board exam questions. In psychiatry especially, these challenges are worsened by fairness and bias issues, since models can be swayed by patient demographics even when those factors should not influence clinical decisions. Thus, we present an expert-created and annotated dataset spanning five critical domains of decision-making in mental healthcare: treatment, diagnosis, documentation, monitoring, and triage. This U.S.-centric dataset - created without any LM assistance - is designed to capture the nuanced clinical reasoning and daily ambiguities mental health practitioners encounter, reflecting the inherent complexities of care delivery that are missing from existing datasets. Almost all base questions with five answer options each have had the decision-irrelevant demographic patient information removed and replaced with variables, e.g., for age or ethnicity, and are available for male, female, or non-binary-coded patients. This design enables systematic evaluations of model performance and bias by studying how demographic factors affect decision-making. For question categories dealing with ambiguity and multiple valid answer options, we create a preference dataset with uncertainties from the expert annotations. We outline a series of intended use cases and demonstrate the usability of our dataset by evaluating sixteen off-the-shelf and six (mental) health fine-tuned LMs on category-specific task accuracy, on the fairness impact of patient demographic information on decision-making, and how consistently free-form responses deviate from human-annotated samples.

翻译：当前医学语言模型（LM）基准测试往往过度简化日常临床实践任务的复杂性，转而依赖多项选择题形式的执业资格考试对LM进行评估。尤其在精神病学领域，公平性与偏见问题加剧了这些挑战，因为模型可能受患者人口统计学特征影响，即使这些因素本不应左右临床决策。为此，我们提出了一个由专家创建并标注的数据集，涵盖心理健康护理中五个关键决策领域：治疗、诊断、文书记录、监测和分诊。这个以美国为中心的数据集——在无任何LM辅助下创建——旨在捕捉心理健康从业者面临的细微临床推理与日常模糊情境，反映现有数据集中缺失的护理服务固有复杂性。几乎所有包含五个选项的基础问题均已移除与决策无关的患者人口统计信息，并替换为变量（如年龄或种族），且提供针对男性、女性或非二元性别编码患者的版本。该设计通过研究人口统计因素如何影响决策，实现了对模型性能与偏见的系统性评估。针对涉及模糊性和多个有效答案选项的问题类别，我们基于专家标注中的不确定性创建了偏好数据集。我们规划了一系列预期使用场景，并通过评估十六个现成模型及六个（心理）健康领域微调LM在特定类别任务准确度、患者人口统计信息对决策的公平性影响，以及自由形式回答与人工标注样本的偏离一致性，验证了数据集的可用性。