Large language models (LLMs) have shown considerable promise in clinical natural language processing, yet few domain-specific datasets exist to rigorously evaluate their performance on radiology tasks. In this work, we introduce an annotated corpus of 6,393 radiology reports from 586 patients, each labeled for follow-up imaging status, to support the development and benchmarking of follow-up adherence detection systems. Using this corpus, we systematically compared traditional machine-learning classifiers, including logistic regression (LR), support vector machines (SVM), Longformer, and a fully fine-tuned Llama3-8B-Instruct, with recent generative LLMs. To evaluate generative LLMs, we tested GPT-4o and the open-source GPT-OSS-20B under two configurations: a baseline (Base) and a task-optimized (Advanced) setting that focused inputs on metadata, recommendation sentences, and their surrounding context. A refined prompt for GPT-OSS-20B further improved reasoning accuracy. Performance was assessed using precision, recall, and F1 scores with 95% confidence intervals estimated via non-parametric bootstrapping. Inter-annotator agreement was high (F1 = 0.846). GPT-4o (Advanced) achieved the best performance (F1 = 0.832), followed closely by GPT-OSS-20B (Advanced; F1 = 0.828). LR and SVM also performed strongly (F1 = 0.776 and 0.775), underscoring that while LLMs approach human-level agreement through prompt optimization, interpretable and resource-efficient models remain valuable baselines.

翻译：大语言模型（LLMs）在临床自然语言处理中展现出巨大潜力，但针对放射学任务严格评估其性能的领域特定数据集仍较为缺乏。本研究引入了一个包含586名患者的6,393份放射学报告的标注语料库，每份报告均标注了随访影像状态，以支持随访依从性检测系统的开发与基准测试。利用该语料库，我们系统比较了传统机器学习分类器（包括逻辑回归（LR）、支持向量机（SVM）、Longformer和完全微调的Llama3-8B-Instruct）与近期生成式大语言模型。为评估生成式大语言模型，我们在两种配置下测试了GPT-4o和开源模型GPT-OSS-20B：基线（Base）设置和任务优化（Advanced）设置，后者将输入聚焦于元数据、建议句及其上下文。针对GPT-OSS-20B优化的提示进一步提升了推理准确性。性能评估采用精确率、召回率和F1分数，并通过非参数自助法估计95%置信区间。标注者间一致性较高（F1 = 0.846）。GPT-4o（Advanced）取得了最佳性能（F1 = 0.832），GPT-OSS-20B（Advanced；F1 = 0.828）紧随其后。LR和SVM也表现强劲（F1分别为0.776和0.775），这表明尽管大语言模型通过提示优化接近人类标注一致性，可解释且资源高效的模型仍是重要的基线参考。