Background: As large language models (LLMs) are increasingly used in healthcare and medical consultation settings, a growing concern is whether these models can respond to medical inquiries in a manner that is ethically compliant--particularly in accordance with local ethical standards. To address the pressing need for comprehensive research on reliability and safety, this study systematically evaluates LLM performance in answering questions related to reproductive ethics, specifically assessing their alignment with Chinese ethical regulations. Methods: We evaluated eight prominent LLMs (e.g., GPT-4, Claude-3.7) on a custom test set of 986 questions (906 subjective, 80 objective) derived from 168 articles within Chinese reproductive ethics regulations. Subjective responses were evaluated using a novel six-dimensional scoring rubric assessing Safety (Normative Compliance, Guidance Safety) and Quality of the Answer (Problem Identification, Citation, Suggestion, Empathy). Results: Significant safety issues were prevalent, with risk rates for unsafe or misleading advice reaching 29.91%. A systemic weakness was observed across all models: universally poor performance in citing normative sources and expressing empathy. We also identified instances of anomalous moral reasoning, including logical self-contradictions and responses violating fundamental moral intuitions. Conclusions: Current LLMs are unreliable and unsafe for autonomous reproductive ethics counseling. Despite knowledge recall, they exhibit critical deficiencies in safety, logical consistency, and essential humanistic skills. These findings serve as a critical cautionary note against premature deployment, urging future development to prioritize robust reasoning, regulatory justification, and empathy.
翻译:背景:随着大型语言模型(LLM)在医疗保健和医疗咨询场景中的应用日益增多,一个日益凸显的关切是这些模型能否以符合伦理——特别是符合当地伦理标准——的方式回应医疗询问。为应对对可靠性与安全性进行全面研究的迫切需求,本研究系统评估了LLM在回答与生殖伦理相关问题时,特别是其与中国伦理法规对齐程度的表现。方法:我们基于中国生殖伦理法规中的168篇文章,构建了一个包含986个问题(906个主观题,80个客观题)的定制测试集,并评估了八个主流LLM(例如GPT-4、Claude-3.7)。主观回答采用一个新颖的六维评分标准进行评估,该标准涵盖安全性(规范性遵从、引导安全性)与回答质量(问题识别、引用、建议、共情)。结果:普遍存在显著的安全性问题,提供不安全或误导性建议的风险率达到29.91%。所有模型均存在一个系统性弱点:在引用规范性来源和表达共情方面表现普遍不佳。我们还发现了异常道德推理的实例,包括逻辑自相矛盾以及违反基本道德直觉的回答。结论:当前的LLM对于自主的生殖伦理咨询而言是不可靠且不安全的。尽管具备知识回忆能力,但它们在安全性、逻辑一致性以及必要的人文技能方面存在严重缺陷。这些发现为过早部署LLM敲响了重要的警钟,并敦促未来的发展应优先考虑稳健的推理、法规依据和共情能力。