Writing radiology reports from medical images requires a high level of domain expertise. It is time-consuming even for trained radiologists and can be error-prone for inexperienced radiologists. It would be appealing to automate this task by leveraging generative AI, which has shown drastic progress in vision and language understanding. In particular, Large Language Models (LLM) have demonstrated impressive capabilities recently and continued to set new state-of-the-art performance on almost all natural language tasks. While many have proposed architectures to combine vision models with LLMs for multimodal tasks, few have explored practical fine-tuning strategies. In this work, we proposed a simple yet effective two-stage fine-tuning protocol to align visual features to LLM's text embedding space as soft visual prompts. Our framework with OpenLLaMA-7B achieved state-of-the-art level performance without domain-specific pretraining. Moreover, we provide detailed analyses of soft visual prompts and attention mechanisms, shedding light on future research directions.

翻译：从医学图像撰写放射学报告需要高水平的领域专业知识。即便是训练有素的放射科医师，这一过程也耗时费力，而对于经验不足的放射科医师而言，则容易出错。利用生成式人工智能自动化此项任务具有吸引力，该技术在视觉和语言理解方面已取得显著进展。特别是，大型语言模型（LLM）近期展现出令人瞩目的能力，并持续在几乎所有自然语言任务中刷新最佳性能记录。尽管许多研究提出了将视觉模型与LLM结合用于多模态任务的架构，但鲜有研究探讨实用的微调策略。在本工作中，我们提出了一种简单而有效的两阶段微调协议，将视觉特征作为软视觉提示与LLM的文本嵌入空间对齐。我们的框架基于OpenLLaMA-7B，在无需领域特定预训练的情况下实现了与当前最佳方法相当的性能。此外，我们对软视觉提示和注意力机制提供了详细分析，为未来研究方向提供了启示。