Serious illness conversations (SICs) align care with patients' values, goals, and preferences, yet they rarely occur in emergency departments (EDs), where time constraints and emotional burden often leave clinicians making high-stakes decisions without documented insight into what matters most to patients. We present a case study of ED GOAL-AI, a voice-based conversational agent for brief, structured values discussions with older adults in the ED, evaluated with 55 patients for feasibility and acceptability. Most participants completed the conversation and reported the interaction as acceptable and feasible, with ratings of feeling heard and understood comparable to clinicians. However, we also observed critical failure modes, including boundary violations such as hallucinated diagnostic statements, highlighting ethical and emotional risks. This work points to early promise for AI-mediated SICs while underscoring the need for careful boundary setting and participatory design before broader deployment.

翻译：严重疾病对话旨在使护理与患者的价值观、目标和偏好相一致，然而这类对话在急诊科中很少发生。在急诊科，时间限制和情感负担常常导致临床医生在缺乏对患者最关心事项的书面洞察的情况下做出高风险决策。本研究通过ED GOAL-AI案例研究——一种面向急诊科老年人进行简短结构化价值观讨论的语音对话代理——在55名患者中评估了其可行性与可接受性。大多数参与者完成了对话，并报告该交互具有可接受性和可行性，其“被倾听和理解”的感受评分与临床医生水平相当。然而，我们也观察到关键的失败模式，包括边界侵犯（例如幻觉性诊断陈述），这突显了伦理和情感风险。本工作揭示了AI介导的严重疾病对话的早期潜力，同时强调了在广泛部署前需要谨慎设定边界并进行参与式设计。