Purpose: This study aims to evaluate the effectiveness of large language models (LLMs) in automating disease annotation of CT radiology reports. We compare a rule-based algorithm (RBA), RadBERT, and three lightweight open-weight LLMs for multi-disease labeling of chest, abdomen, and pelvis (CAP) CT reports. Materials and Methods: This retrospective study analyzed 40,833 chest-abdomen-pelvis (CAP) CT reports from 29,540 patients, with 1,789 reports manually annotated across three organ systems. External validation was conducted using the CT RATE dataset. Three open-weight LLMs were tested with zero-shot prompting. Performance was evaluated using Cohen's Kappa ($κ$) and micro/macro-averaged F1 scores. Results: In the internal test set of 12,197 CAP reports from 8,854 patients, Llama-3.1 8B and Gemma-3 27B showed the highest agreement ($κ$ median: 0.87). On the manually annotated set, Gemma-3 27B achieved the top macro-F1 (0.82), followed by Llama-3.1 8B (0.79), while the RBA scored lowest (0.64). On the CT RATE dataset (lungs/pleura labels only), Llama-3.1 8B performed best (0.91), with Gemma-3 27B close behind (0.89). Performance differences were mainly due to differing labeling practices, especially for labels with high subjectivity such as atelectasis. Conclusion: Lightweight LLMs outperform rule-based methods for CT report annotation and generalize across organ systems with zero-shot prompting. However, binary labels alone cannot capture the full nuance of report language. LLMs can provide a flexible, efficient solution aligned with clinical judgment and user needs.

翻译：目的：本研究旨在评估大语言模型在自动化CT放射学报告疾病标注方面的有效性。我们比较了基于规则的算法、RadBERT以及三种轻量级开源权重LLM在胸部、腹部和盆腔CT报告的多疾病标注任务上的表现。材料与方法：这项回顾性研究分析了来自29,540名患者的40,833份胸腹盆腔CT报告，其中1,789份报告在三个器官系统上进行了人工标注。使用CT RATE数据集进行了外部验证。测试了三种开源权重LLM的零样本提示性能。使用Cohen's Kappa ($κ$) 和微观/宏观平均F1分数评估性能。结果：在来自8,854名患者的12,197份CAP报告的内部测试集中，Llama-3.1 8B和Gemma-3 27B表现出最高的一致性（$κ$中位数：0.87）。在人工标注集上，Gemma-3 27B获得了最高的宏观F1分数（0.82），其次是Llama-3.1 8B（0.79），而RBA得分最低（0.64）。在CT RATE数据集（仅肺/胸膜标签）上，Llama-3.1 8B表现最佳（0.91），Gemma-3 27B紧随其后（0.89）。性能差异主要源于标注实践的差异，特别是对于肺不张等高主观性标签。结论：轻量级LLM在CT报告标注任务上优于基于规则的方法，并能通过零样本提示泛化到不同器官系统。然而，仅使用二元标签无法完全捕捉报告语言的细微差别。LLM可以提供一种灵活、高效的解决方案，与临床判断和用户需求保持一致。