With increasing reliance on medical imaging in clinical practices, automated report generation from medical images is in great demand. Existing report generation methods typically adopt an encoder-decoder deep learning framework to build a uni-directional image-to-report mapping. However, such a framework ignores the bi-directional mutual associations between images and reports, thus incurring difficulties in associating the intrinsic medical meanings between them. Recent generative representation learning methods have demonstrated the benefits of dual-modal learning from both image and text modalities. However, these methods exhibit two major drawbacks for medical report generation: 1) they tend to capture morphological information and have difficulties in capturing subtle pathological semantic information, and 2) they predict masked text rely on both unmasked images and text, inevitably degrading performance when inference is based solely on images. In this study, we propose a new report generation framework with dual-modal dynamic traceback learning (DTrace) to overcome the two identified drawbacks and enable dual-modal learning for medical report generation. To achieve this, our DTrace introduces a traceback mechanism to control the semantic validity of generated content via self-assessment. Further, our DTrace introduces a dynamic learning strategy to adapt to various proportions of image and text input, enabling report generation without reliance on textual input during inference. Extensive experiments on two well-benchmarked datasets (IU-Xray and MIMIC-CXR) show that our DTrace outperforms state-of-the-art medical report generation methods.
翻译:随着临床实践中医学影像的日益依赖,从医学图像自动生成报告的需求十分迫切。现有报告生成方法通常采用编码器-解码器深度学习框架构建单向的图像到报告映射。然而,这种框架忽略了图像与报告之间的双向互关联,因此难以关联它们之间的内在医学语义。近年来,生成式表示学习方法展示了从图像和文本双模态学习的优势。然而,这些方法在医学报告生成中存在两个主要缺陷:1)它们倾向于捕捉形态学信息,难以捕捉细微的病理语义信息;2)它们在预测掩码文本时依赖于未被掩码的图像和文本,当推理仅基于图像时不可避免地导致性能下降。在本研究中,我们提出了一种带有双模态动态回溯学习(DTrace)的新型报告生成框架,以克服这两个缺陷并实现用于医学报告生成的双模态学习。为此,我们的DTrace引入了一种回溯机制,通过自评估来控制生成内容的语义有效性。此外,DTrace引入了一种动态学习策略,以适应图像和文本输入的不同比例,从而在推理时无需依赖文本输入即可生成报告。在两个广泛使用的基准数据集(IU-Xray和MIMIC-CXR)上进行的大量实验表明,我们的DTrace优于最先进的医学报告生成方法。