The capabilities of large language models (LLMs) have been progressing at a breathtaking speed, leaving even their own developers grappling with the depth of their potential and risks. While initial steps have been taken to evaluate the safety and alignment of general-knowledge LLMs, exposing some weaknesses, to our knowledge, the safety and alignment of medical LLMs has not been evaluated despite their risks for personal health and safety, public health and safety, and human rights. To this end, we carry out the first safety evaluation for medical LLMs. Specifically, we set forth a definition of medical safety and alignment for medical artificial intelligence systems, develop a dataset of harmful medical questions to evaluate the medical safety and alignment of an LLM, evaluate both general and medical safety and alignment of medical LLMs, demonstrate fine-tuning as an effective mitigation strategy, and discuss broader, large-scale approaches used by the machine learning community to develop safe and aligned LLMs. We hope that this work casts light on the safety and alignment of medical LLMs and motivates future work to study it and develop additional mitigation strategies, minimizing the risks of harm of LLMs in medicine.

翻译：大语言模型（LLM）的能力正以惊人的速度发展，甚至其开发者自身也难以完全把握其潜力与风险。虽然针对通用知识型LLM的安全性与对齐性评估已初步展开并暴露出部分弱点，但据我们所知，尽管医学LLM可能对个人健康安全、公共卫生安全及人权构成威胁，其安全性与对齐性尚未得到系统评估。为此，我们开展了首次针对医学LLM的安全性评估。具体而言，我们定义了医学人工智能系统的医疗安全与对齐标准；构建了有害医学问题数据集以评估LLM的医疗安全性与对齐性；全面评估了医学LLM在通用领域与医学领域的安全性与对齐性；验证了微调作为有效缓解策略的可行性；并探讨了机器学习社区用于开发安全与对齐LLM的更广泛规模化方法。我们希望本研究能揭示医学LLM的安全性与对齐性问题，推动后续研究及更多缓解策略的制定，从而最大限度降低LLM在医学领域的潜在危害风险。