Diabetic Retinopathy (DR), a prevalent and severe complication of diabetes, affects millions of individuals globally, underscoring the need for accurate and timely diagnosis. Recent advancements in imaging technologies, such as Ultra-WideField Color Fundus Photography (UWF-CFP) imaging and Optical Coherence Tomography Angiography (OCTA), provide opportunities for the early detection of DR but also pose significant challenges given the disparate nature of the data they produce. This study introduces a novel multimodal approach that leverages these imaging modalities to notably enhance DR classification. Our approach integrates 2D UWF-CFP images and 3D high-resolution 6x6 mm$^3$ OCTA (both structure and flow) images using a fusion of ResNet50 and 3D-ResNet50 models, with Squeeze-and-Excitation (SE) blocks to amplify relevant features. Additionally, to increase the model's generalization capabilities, a multimodal extension of Manifold Mixup, applied to concatenated multimodal features, is implemented. Experimental results demonstrate a remarkable enhancement in DR classification performance with the proposed multimodal approach compared to methods relying on a single modality only. The methodology laid out in this work holds substantial promise for facilitating more accurate, early detection of DR, potentially improving clinical outcomes for patients.

翻译：糖尿病视网膜病变（DR）作为糖尿病常见且严重的并发症，影响着全球数百万患者，凸显了准确及时诊断的必要性。超广角彩色眼底照相（UWF-CFP）及光学相干断层扫描血管成像（OCTA）等影像技术的进展，为DR早期检测提供了新契机，但鉴于这些数据在本质上的异质性，也带来了显著挑战。本研究提出一种创新的多模态方法，通过整合上述影像模态显著提升DR分类性能。该方法融合二维UWF-CFP图像与三维高分辨率6×6 mm³ OCTA（结构和血流）图像，采用ResNet50与3D-ResNet50模型混合架构，并引入挤压-激励（SE）模块强化相关特征。此外，为增强模型泛化能力，本研究实现了应用于拼接多模态特征的多模态流形混合（Manifold Mixup）扩展。实验结果表明，与仅依赖单一模态的方法相比，所提出的多模态方法在DR分类性能上实现了显著提升。本研究提出的方法在促进更精准、更早期的DR检测方面具有重要潜力，有望改善患者的临床预后。