Medical image interpretation is central to most clinical applications such as disease diagnosis, treatment planning, and prognostication. In clinical practice, radiologists examine medical images and manually compile their findings into reports, which can be a time-consuming process. Automated approaches to radiology report generation, therefore, can reduce radiologist workload and improve efficiency in the clinical pathway. While recent deep-learning approaches for automated report generation from medical images have seen some success, most studies have relied on image-derived features alone, ignoring non-imaging patient data. Although a few studies have included the word-level contexts along with the image, the use of patient demographics is still unexplored. This paper proposes a novel multi-modal transformer network that integrates chest x-ray (CXR) images and associated patient demographic information, to synthesise patient-specific radiology reports. The proposed network uses a convolutional neural network to extract visual features from CXRs and a transformer-based encoder-decoder network that combines the visual features with semantic text embeddings of patient demographic information, to synthesise full-text radiology reports. Data from two public databases were used to train and evaluate the proposed approach. CXRs and reports were extracted from the MIMIC-CXR database and combined with corresponding patients' data MIMIC-IV. Based on the evaluation metrics used including patient demographic information was found to improve the quality of reports generated using the proposed approach, relative to a baseline network trained using CXRs alone. The proposed approach shows potential for enhancing radiology report generation by leveraging rich patient metadata and combining semantic text embeddings derived thereof, with medical image-derived visual features.

翻译：医学图像解读是大多数临床应用的核心，例如疾病诊断、治疗计划和预后评估。在临床实践中，放射科医生检查医学图像并手动将结果整理成报告，这一过程可能非常耗时。因此，自动化放射学报告生成方法可以减轻放射科医生的工作负担，并提高临床路径的效率。尽管近期基于深度学习的医学图像自动报告生成方法取得了一定成功，但大多数研究仅依赖图像衍生特征，忽略了非影像患者数据。虽然少数研究将词级上下文与图像结合，但患者人口统计学信息的使用仍未得到探索。本文提出了一种新颖的多模态Transformer网络，该网络整合了胸部X光片（CXR）图像及相关患者人口统计学信息，以合成患者特异性放射学报告。所提出的网络使用卷积神经网络从CXR中提取视觉特征，并采用基于Transformer的编码器-解码器网络，将视觉特征与患者人口统计学信息的语义文本嵌入相结合，以合成全文本放射学报告。使用来自两个公共数据库的数据对所提出的方法进行训练和评估。从MIMIC-CXR数据库中提取CXR和报告，并与对应的患者数据MIMIC-IV相结合。基于评估指标，发现纳入患者人口统计学信息可提高使用所提出方法生成的报告质量，相对于仅使用CXR训练的基线网络。所提出的方法通过利用丰富的患者元数据并结合由此衍生的语义文本嵌入与医学图像衍生的视觉特征，展示了增强放射学报告生成的潜力。