This study investigates the effects of including patients' clinical information on the performance of deep learning (DL) classifiers for disease location in chest X-ray images. Although current classifiers achieve high performance using chest X-ray images alone, our interviews with radiologists indicate that clinical data is highly informative and essential for interpreting images and making proper diagnoses. In this work, we propose a novel architecture consisting of two fusion methods that enable the model to simultaneously process patients' clinical data (structured data) and chest X-rays (image data). Since these data modalities are in different dimensional spaces, we propose a spatial arrangement strategy, spatialization, to facilitate the multimodal learning process in a Mask R-CNN model. We performed an extensive experimental evaluation using MIMIC-Eye, a dataset comprising modalities: MIMIC-CXR (chest X-ray images), MIMIC IV-ED (patients' clinical data), and REFLACX (annotations of disease locations in chest X-rays). Results show that incorporating patients' clinical data in a DL model together with the proposed fusion methods improves the disease localization in chest X-rays by 12\% in terms of Average Precision compared to a standard Mask R-CNN using only chest X-rays. Further ablation studies also emphasize the importance of multimodal DL architectures and the incorporation of patients' clinical data in disease localization. The architecture proposed in this work is publicly available to promote the scientific reproducibility of our study (https://github.com/ChihchengHsieh/multimodal-abnormalities-detection)

翻译：本研究探究了将患者临床信息纳入深度学习分类器对胸部X射线图像病灶定位性能的影响。尽管现有分类器仅凭胸部X射线图像即可实现较高性能，但我们与放射科医师的访谈表明，临床数据具有极高的信息价值，且对图像解读与正确诊断至关重要。本文提出一种包含两种融合方法的新型架构，使模型能够同时处理患者的临床数据（结构化数据）与胸部X射线（图像数据）。由于这两种数据模态处于不同维度空间，我们提出了一种空间排列策略——空间化处理，以促进Mask R-CNN模型中的多模态学习过程。我们使用MIMIC-Eye数据集进行了广泛的实验评估，该数据集包含三种模态：MIMIC-CXR（胸部X射线图像）、MIMIC IV-ED（患者临床数据）以及REFLACX（胸部X射线病灶位置标注）。结果表明，在深度学习模型中融入患者临床数据并采用所提出的融合方法后，在平均精度指标上，病灶定位性能相较于仅使用胸部X射线的标准Mask R-CNN提升了12%。进一步的消融研究也强调了多模态深度学习架构及纳入患者临床数据在病灶定位中的重要性。本研究所提出的架构已公开，以促进研究的可复现性（https://github.com/ChihchengHsieh/multimodal-abnormalities-detection）。