Large language models (LLMs) excel on multiple-choice clinical diagnosis benchmarks, yet it is unclear how much of this performance reflects underlying probabilistic reasoning. We study this through questions from MedQA, where the task is to select the most likely diagnosis. We introduce the Frequency-Based Probabilistic Ranker (FBPR), a lightweight method that scores options with a smoothed Naive Bayes over concept-diagnosis co-occurrence statistics from a large corpus. When co-occurrence statistics were sourced from the pretraining corpora for OLMo and Llama, FBPR achieves comparable performance to the corresponding LLMs pretrained on that same corpus. Direct LLM inference and FBPR largely get different questions correct, with an overlap only slightly above random chance, indicating complementary strengths of each method. These findings highlight the continued value of explicit probabilistic baselines: they provide a meaningful performance reference point and a complementary signal for potential hybridization. While the performance of LLMs seems to be driven by a mechanism other than simple frequency aggregation, we show that an approach similar to the historically grounded, low-complexity expert systems still accounts for a substantial portion of benchmark performance.

翻译：大型语言模型（LLMs）在多项选择临床诊断基准测试中表现出色，但其性能在多大程度上反映了底层的概率推理能力尚不明确。本研究通过MedQA中的问题对此进行探讨，任务目标为选择最可能的诊断。我们引入了基于频率的概率排序器（FBPR），这是一种轻量级方法，通过基于大型语料库中概念-诊断共现统计的平滑朴素贝叶斯算法对选项进行评分。当共现统计数据来源于OLMo和Llama的预训练语料时，FBPR在相同语料上达到了与对应预训练LLMs相当的性能。直接LLM推理与FBPR在正确回答的问题上存在显著差异，其重叠率仅略高于随机概率，表明两种方法具有互补优势。这些发现凸显了显式概率基线的持续价值：它们提供了有意义的性能参照点，并为潜在混合方法提供了互补信号。尽管LLMs的性能似乎由超越简单频率聚合的机制驱动，但我们证明，一种类似历史上基于低复杂度专家系统的方法，仍然在基准测试性能中占据重要部分。