Reinforcement learning-based large language models, such as ChatGPT, are believed to have potential to aid human experts in many domains, including healthcare. There is, however, little work on ChatGPT's ability to perform a key task in healthcare: formal, probabilistic medical diagnostic reasoning. This type of reasoning is used, for example, to update a pre-test probability to a post-test probability. In this work, we probe ChatGPT's ability to perform this task. In particular, we ask ChatGPT to give examples of how to use Bayes rule for medical diagnosis. Our prompts range from queries that use terminology from pure probability (e.g., requests for a "posterior probability") to queries that use terminology from the medical diagnosis literature (e.g., requests for a "post-test probability"). We show how the introduction of medical variable names leads to an increase in the number of errors that ChatGPT makes. Given our results, we also show how one can use prompt engineering to facilitate ChatGPT's partial avoidance of these errors. We discuss our results in light of recent commentaries on sensitivity and specificity. We also discuss how our results might inform new research directions for large language models.

翻译：基于强化学习的大型语言模型（如ChatGPT）被认为具有在包括医疗保健在内的多个领域辅助人类专家的潜力。然而，目前很少有研究探讨ChatGPT执行医疗保健领域关键任务的能力：形式化的概率医学诊断推理。例如，这类推理可用于将验前概率更新为验后概率。在本研究中，我们探究了ChatGPT执行此任务的能力。具体而言，我们要求ChatGPT举例说明如何使用贝叶斯定理进行医学诊断。我们的提示从使用纯概率术语（如请求"后验概率"）到使用医学诊断文献术语（如请求"验后概率"）不等。我们展示了引入医学变量名称如何导致ChatGPT错误数量的增加。基于我们的结果，我们还展示了如何通过提示工程促使ChatGPT部分避免这些错误。我们结合近期关于敏感性和特异性的评述讨论了研究结果，同时探讨了这些结果如何为大型语言模型的新研究方向提供启示。