Adapting foundation models for medical image analysis requires finetuning them on a considerable amount of data because of extreme distribution shifts between natural (source) data used for pretraining and medical (target) data. However, collecting task-specific medical data for such finetuning at a central location raises many privacy concerns. Although Federated learning (FL) provides an effective means for training on private decentralized data, communication costs in federating large foundation models can quickly become a significant bottleneck, impacting the solution's scalability. In this work, we address this problem of efficient communication while ensuring effective learning in FL by combining the strengths of Parameter-Efficient Fine-tuning (PEFT) with FL. Specifically, we study plug-and-play Low-Rank Adapters (LoRA) in a federated manner to adapt the Segment Anything Model (SAM) for 3D medical image segmentation. Unlike prior works that utilize LoRA and finetune the entire decoder, we critically analyze the contribution of each granular component of SAM on finetuning performance. Thus, we identify specific layers to be federated that are very efficient in terms of communication cost while producing on-par accuracy. Our experiments show that retaining the parameters of the SAM model (including most of the decoder) in their original state during adaptation is beneficial because fine-tuning on small datasets tends to distort the inherent capabilities of the underlying foundation model. On Fed-KiTS, our approach decreases communication cost (~48x) compared to full fine-tuning while increasing performance (~6% Dice score) in 3D segmentation tasks. Our approach performs similar to SAMed while achieving ~2.8x reduction in communication and parameters to be finetuned. We further validate our approach with experiments on Fed-IXI and Prostate MRI datasets.

翻译：将基础模型应用于医学图像分析时，由于预训练使用的自然（源）数据与医学（目标）数据之间存在极大的分布偏移，通常需要大量数据进行微调。然而，在中心化位置收集此类任务特定的医学数据会引发诸多隐私问题。尽管联邦学习（FL）为在私有分散数据上进行训练提供了有效手段，但联邦化大型基础模型时的通信成本可能迅速成为显著瓶颈，影响解决方案的可扩展性。在本研究中，我们通过结合参数高效微调（PEFT）与联邦学习的优势，解决了在确保联邦学习中有效学习的同时实现高效通信的问题。具体而言，我们以联邦方式研究即插即用的低秩适配器（LoRA），用于使Segment Anything Model（SAM）适应三维医学图像分割任务。与先前利用LoRA并微调整个解码器的工作不同，我们批判性地分析了SAM各粒度组件对微调性能的贡献。因此，我们确定了需要联邦化的特定层，这些层在通信成本方面非常高效，同时能产生相当的精度。我们的实验表明，在适应过程中保持SAM模型参数（包括大部分解码器）的原始状态是有益的，因为在小数据集上进行微调往往会扭曲底层基础模型的固有能力。在Fed-KiTS数据集上，我们的方法相比全参数微调降低了通信成本（约48倍），同时在三维分割任务中提升了性能（约6% Dice分数）。我们的方法在达到与SAMed相当性能的同时，实现了约2.8倍的通信及待微调参数量的减少。我们通过在Fed-IXI和前列腺MRI数据集上的实验进一步验证了该方法的有效性。