Large Language Models (LLMs) are revolutionizing medical diagnostics by enhancing both disease classification and clinical decision-making. In this study, we evaluate the performance of two LLM- based diagnostic tools, DeepSeek R1 and O3 Mini, using a structured dataset of symptoms and diagnoses. We assessed their predictive accuracy at both the disease and category levels, as well as the reliability of their confidence scores. DeepSeek R1 achieved a disease-level accuracy of 76% and an overall accuracy of 82%, outperforming O3 Mini, which attained 72% and 75% respectively. Notably, DeepSeek R1 demonstrated exceptional performance in Mental Health, Neurological Disorders, and Oncology, where it reached 100% accuracy, while O3 Mini excelled in Autoimmune Disease classification with 100% accuracy. Both models, however, struggled with Respiratory Disease classification, recording accuracies of only 40% for DeepSeek R1 and 20% for O3 Mini. Additionally, the analysis of confidence scores revealed that DeepSeek R1 provided high-confidence predictions in 92% of cases, compared to 68% for O3 Mini. Ethical considerations regarding bias, model interpretability, and data privacy are also discussed to ensure the responsible integration of LLMs into clinical practice. Overall, our findings offer valuable insights into the strengths and limitations of LLM-based diagnostic systems and provide a roadmap for future enhancements in AI-driven healthcare.

翻译：大型语言模型（LLMs）正通过提升疾病分类与临床决策能力，革新医学诊断领域。本研究利用结构化症状与诊断数据集，评估了两种基于LLM的诊断工具——DeepSeek R1与O3 Mini的性能。我们分别从疾病层面和类别层面评估了其预测准确性，并检验了其置信度评分的可靠性。DeepSeek R1在疾病层面准确率达到76%，整体准确率为82%，优于O3 Mini的72%和75%。值得注意的是，DeepSeek R1在心理健康、神经系统疾病和肿瘤学领域表现卓越，准确率达到100%，而O3 Mini在自身免疫性疾病分类中实现了100%的准确率。然而，两种模型在呼吸系统疾病分类中均表现不佳，DeepSeek R1准确率仅为40%，O3 Mini仅为20%。此外，置信度分析显示，DeepSeek R1在92%的病例中提供了高置信度预测，而O3 Mini的这一比例为68%。本文还讨论了关于偏见、模型可解释性和数据隐私的伦理考量，以确保LLMs在临床实践中的负责任整合。总体而言，我们的研究结果为基于LLM的诊断系统的优势与局限性提供了重要见解，并为未来人工智能驱动医疗的改进指明了方向。