The use of machine learning (ML) for cancer staging through medical image analysis has gained substantial interest across medical disciplines. When accompanied by the innovative federated learning (FL) framework, ML techniques can further overcome privacy concerns related to patient data exposure. Given the frequent presence of diverse data modalities within patient records, leveraging FL in a multi-modal learning framework holds considerable promise for cancer staging. However, existing works on multi-modal FL often presume that all data-collecting institutions have access to all data modalities. This oversimplified approach neglects institutions that have access to only a portion of data modalities within the system. In this work, we introduce a novel FL architecture designed to accommodate not only the heterogeneity of data samples, but also the inherent heterogeneity/non-uniformity of data modalities across institutions. We shed light on the challenges associated with varying convergence speeds observed across different data modalities within our FL system. Subsequently, we propose a solution to tackle these challenges by devising a distributed gradient blending and proximity-aware client weighting strategy tailored for multi-modal FL. To show the superiority of our method, we conduct experiments using The Cancer Genome Atlas program (TCGA) datalake considering different cancer types and three modalities of data: mRNA sequences, histopathological image data, and clinical information.

翻译：利用机器学习通过医学图像分析进行癌症分期已在多个医学领域引起广泛关注。结合创新的联邦学习框架，机器学习技术可进一步克服患者数据暴露相关的隐私问题。鉴于患者记录中常存在多种数据模态，在多模态学习框架中利用联邦学习对癌症分期具有重要前景。然而，现有关于多模态联邦学习的研究通常假设所有数据收集机构都能访问全部数据模态。这种过于简化的方法忽略了系统中仅能访问部分数据模态的机构。本文提出一种新颖的联邦学习架构，既能适应数据样本的异质性，又能处理各机构间数据模态固有的异质性/非均匀性。我们揭示了联邦学习系统中不同数据模态收敛速度差异带来的挑战，并提出相应解决方案：设计一种针对多模态联邦学习的分布式梯度混合与邻近感知客户端加权策略。为证明本方法的优越性，我们利用癌症基因组图谱(TCGA)数据库，针对不同癌症类型及mRNA序列、组织病理图像数据和临床信息三种数据模态开展实验。