Image-to-text radiology report generation aims to automatically produce radiology reports that describe the findings in medical images. Most existing methods focus solely on the image data, disregarding the other patient information accessible to radiologists. In this paper, we present a novel multi-modal deep neural network framework for generating chest X-rays reports by integrating structured patient data, such as vital signs and symptoms, alongside unstructured clinical notes.We introduce a conditioned cross-multi-head attention module to fuse these heterogeneous data modalities, bridging the semantic gap between visual and textual data. Experiments demonstrate substantial improvements from using additional modalities compared to relying on images alone. Notably, our model achieves the highest reported performance on the ROUGE-L metric compared to relevant state-of-the-art models in the literature. Furthermore, we employed both human evaluation and clinical semantic similarity measurement alongside word-overlap metrics to improve the depth of quantitative analysis. A human evaluation, conducted by a board-certified radiologist, confirms the model's accuracy in identifying high-level findings, however, it also highlights that more improvement is needed to capture nuanced details and clinical context.

翻译：图像到文本的放射学报告生成旨在自动生成描述医学影像中发现的放射学报告。现有方法大多仅关注图像数据，忽视了放射科医生可获取的其他患者信息。本文提出了一种新型多模态深度神经网络框架，通过整合结构化患者数据（如生命体征和症状）与非结构化临床笔记，实现胸部X光报告生成。我们引入了条件交叉多头注意力模块来融合这些异构数据模态，弥合视觉与文本数据之间的语义鸿沟。实验表明，相较于仅依赖图像，使用额外模态能带来显著改进。值得注意的是，与文献中相关的最先进模型相比，我们的模型在ROUGE-L指标上取得了最高报告性能。此外，在词重叠指标之外，我们还采用了人工评估和临床语义相似度测量来提升定量分析的深度。由委员会认证放射科医生执行的人工评估证实了模型在识别高层次发现方面的准确性，但同时也指出，在捕捉细微细节和临床背景方面仍需进一步改进。