Bone health studies are crucial in medical practice for the early detection and treatment of Osteopenia and Osteoporosis. Clinicians usually make a diagnosis based on densitometry (DEXA scans) and patient history. The applications of AI in this field are ongoing research. Most successful methods rely on deep learning models that use vision alone (DEXA/X-ray imagery) and focus on prediction accuracy, while explainability is often disregarded and left to post hoc assessments of input contributions. We propose ProtoMedX, a multi-modal (multimodal) model that uses both DEXA scans of the lumbar spine and patient records. ProtoMedX's prototype-based architecture is explainable by design, which is crucial for medical applications, especially in the context of the upcoming EU AI Act, as it allows explicit analysis of model decisions, including incorrect ones. ProtoMedX demonstrates state-of-the-art performance in bone health classification while also providing explanations that can be visually understood by clinicians. Using a dataset of 4,160 real NHS patients, the proposed ProtoMedX achieves 87.58% accuracy in vision-only tasks and 89.8% in its multi-modal variant, both surpassing existing published methods.

翻译：骨健康研究在医学实践中对于骨质疏松症和骨量减少的早期检测与治疗至关重要。临床医生通常基于骨密度测定（DEXA扫描）和患者病史进行诊断。人工智能在该领域的应用正处于持续研究阶段。目前最成功的方法主要依赖于仅使用视觉信息（DEXA/X射线影像）的深度学习模型，并侧重于预测准确性，而可解释性常被忽视，通常仅通过事后评估输入贡献来实现。我们提出了ProtoMedX，一种同时利用腰椎DEXA扫描和患者病历的多模态模型。ProtoMedX基于原型设计的架构本身具备可解释性，这对于医疗应用至关重要，特别是在即将实施的欧盟人工智能法案背景下，因为它支持对模型决策（包括错误决策）进行显式分析。ProtoMedX在骨健康分类任务中展示了最先进的性能，同时提供临床医生可直观理解的解释。基于包含4,160名真实NHS患者的数据集，所提出的ProtoMedX在纯视觉任务中达到87.58%的准确率，在多模态版本中达到89.8%的准确率，均超越现有已发表方法。