The integration of large language models (LLMs) into the medical field has gained significant attention due to their promising accuracy in simulated clinical decision-making settings. However, clinical decision-making is more complex than simulations because physicians' decisions are shaped by many factors, including the presence of cognitive bias. However, the degree to which LLMs are susceptible to the same cognitive biases that affect human clinicians remains unexplored. Our hypothesis posits that when LLMs are confronted with clinical questions containing cognitive biases, they will yield significantly less accurate responses compared to the same questions presented without such biases.In this study, we developed BiasMedQA, a novel benchmark for evaluating cognitive biases in LLMs applied to medical tasks. Using BiasMedQA we evaluated six LLMs, namely GPT-4, Mixtral-8x70B, GPT-3.5, PaLM-2, Llama 2 70B-chat, and the medically specialized PMC Llama 13B. We tested these models on 1,273 questions from the US Medical Licensing Exam (USMLE) Steps 1, 2, and 3, modified to replicate common clinically-relevant cognitive biases. Our analysis revealed varying effects for biases on these LLMs, with GPT-4 standing out for its resilience to bias, in contrast to Llama 2 70B-chat and PMC Llama 13B, which were disproportionately affected by cognitive bias. Our findings highlight the critical need for bias mitigation in the development of medical LLMs, pointing towards safer and more reliable applications in healthcare.

翻译：大语言模型（LLMs）在医学领域的整合因其在模拟临床决策场景中展现出的令人瞩目的准确性而备受关注。然而，临床决策比模拟更为复杂，因为医生的决策受到包括认知偏差在内的多种因素影响。目前，LLMs是否易受同样影响人类临床医生的认知偏差影响仍属未知。我们假设，当LLMs面对包含认知偏差的临床问题时，其回答准确性将显著低于不含此类偏差的相同问题。在本研究中，我们开发了BiasMedQA这一新型基准，用于评估LLMs在医学任务中的认知偏差。利用BiasMedQA，我们评估了六种LLMs：GPT-4、Mixtral-8x70B、GPT-3.5、PaLM-2、Llama 2 70B-chat以及医学专用模型PMC Llama 13B。我们在来自美国医师资格考试（USMLE）第1、2、3阶段的1,273道考题上测试了这些模型，并对题目进行了修改以复现临床相关的常见认知偏差。分析显示，不同偏差对LLMs的影响程度各异，其中GPT-4对偏差具有显著抵抗力，而Llama 2 70B-chat和PMC Llama 13B则受到认知偏差的显著影响。我们的研究结果凸显了在医学LLMs开发中缓解偏差的紧迫需求，旨在为医疗领域提供更安全、更可靠的应用方案。