In radiology, Artificial Intelligence (AI) has significantly advanced report generation, but automatic evaluation of these AI-produced reports remains challenging. Current metrics, such as Conventional Natural Language Generation (NLG) and Clinical Efficacy (CE), often fall short in capturing the semantic intricacies of clinical contexts or overemphasize clinical details, undermining report clarity. To overcome these issues, our proposed method synergizes the expertise of professional radiologists with Large Language Models (LLMs), like GPT-3.5 and GPT-4 1. Utilizing In-Context Instruction Learning (ICIL) and Chain of Thought (CoT) reasoning, our approach aligns LLM evaluations with radiologist standards, enabling detailed comparisons between human and AI generated reports. This is further enhanced by a Regression model that aggregates sentence evaluation scores. Experimental results show that our "Detailed GPT-4 (5-shot)" model achieves a 0.48 score, outperforming the METEOR metric by 0.19, while our "Regressed GPT-4" model shows even greater alignment with expert evaluations, exceeding the best existing metric by a 0.35 margin. Moreover, the robustness of our explanations has been validated through a thorough iterative strategy. We plan to publicly release annotations from radiology experts, setting a new standard for accuracy in future assessments. This underscores the potential of our approach in enhancing the quality assessment of AI-driven medical reports.

翻译：在放射学领域，人工智能（AI）已显著推进报告生成技术，但如何自动评估这些AI生成报告仍面临挑战。当前指标，如传统自然语言生成（NLG）指标和临床效能（CE）指标，往往无法捕捉临床语境中的语义复杂性，或过度强调临床细节，从而损害报告清晰度。为解决这些问题，我们提出的方法将专业放射科医生的专业知识与大型语言模型（LLMs）（如GPT-3.5和GPT-4）协同运用。通过上下文指令学习（ICIL）和思维链（CoT）推理，我们的方法使LLM评估与放射科医生标准对齐，实现对人类与AI生成报告的精细比对。该方法通过聚合句子评估分数的回归模型进一步增强。实验结果表明，我们的"Detailed GPT-4 (5-shot)"模型得分为0.48，比METEOR指标高出0.19；而"Regressed GPT-4"模型与专家评估的一致性更高，超出最佳现有指标0.35。此外，我们通过彻底的迭代策略验证了解释的鲁棒性。我们计划公开发布放射学专家的标注数据，为未来评估准确性设立新标准。这突显了本方法在提升AI驱动医学报告质量评估方面的潜力。