Large Language Models (LLMs) have demonstrated promise in medical knowledge assessments, yet their practical utility in real-world clinical decision-making remains underexplored. In this study, we evaluated the performance of three state-of-the-art LLMs-ChatGPT-4o, Gemini 1.5 Pro, and LIama 3.3 70B-in clinical decision support across the entire clinical reasoning workflow of a typical patient encounter. Using 36 case studies, we first assessed LLM's out-of-the-box performance across five key sequential clinical decision-making tasks under two temperature settings (default vs. zero): differential diagnosis, essential immediate steps, relevant diagnostic testing, final diagnosis, and treatment recommendation. All models showed high variability by task, achieving near-perfect accuracy in final diagnosis, poor performance in relevant diagnostic testing, and moderate performance in remaining tasks. Furthermore, ChatGPT performed better under the zero temperature, whereas LIama showed stronger performance under the default temperature. Next, we assessed whether prompt engineering could enhance LLM performance by applying variations of the MedPrompt framework, incorporating targeted and random dynamic few-shot learning. The results demonstrate that prompt engineering is not a one-size-fit-all solution. While it significantly improved the performance on the task with lowest baseline accuracy (relevant diagnostic testing), it was counterproductive for others. Another key finding was that the targeted dynamic few-shot prompting did not consistently outperform random selection, indicating that the presumed benefits of closely matched examples may be counterbalanced by loss of broader contextual diversity. These findings suggest that the impact of prompt engineering is highly model and task-dependent, highlighting the need for tailored, context-aware strategies for integrating LLMs into healthcare.

翻译：大语言模型在医学知识评估中展现出潜力，但其在真实世界临床决策中的实际效用仍未得到充分探索。本研究评估了三种前沿大语言模型——ChatGPT-4o、Gemini 1.5 Pro和LIama 3.3 70B——在典型患者就诊全流程临床推理中的临床决策支持表现。基于36个案例研究，我们首先评估了模型在两种温度设置（默认温度与零温度）下，针对五个关键顺序临床决策任务的即用性能：鉴别诊断、必要紧急措施、相关诊断检测、最终诊断及治疗建议。所有模型均表现出明显的任务间性能差异：在最终诊断任务中达到接近完美的准确率，在相关诊断检测任务中表现欠佳，其余任务表现中等。此外，ChatGPT在零温度设置下表现更优，而LIama在默认温度下性能更强。随后，我们通过应用MedPrompt框架的多种变体（包含定向与随机动态少样本学习）评估提示工程能否提升模型性能。结果表明提示工程并非普适性解决方案：虽然它能显著提升基线准确率最低任务（相关诊断检测）的表现，但对其他任务却产生负面效果。另一关键发现是：定向动态少样本提示并未持续优于随机选择，这表明紧密匹配示例的预设优势可能被更广泛语境多样性的缺失所抵消。这些发现提示提示工程的影响高度依赖于模型与具体任务，凸显了将大语言模型整合至医疗领域时需要定制化、情境感知的策略。