The scarcity of well-annotated medical datasets requires leveraging transfer learning from broader datasets like ImageNet or pre-trained models like CLIP. Model soups averages multiple fine-tuned models aiming to improve performance on In-Domain (ID) tasks and enhance robustness against Out-of-Distribution (OOD) datasets. However, applying these methods to the medical imaging domain faces challenges and results in suboptimal performance. This is primarily due to differences in error surface characteristics that stem from data complexities such as heterogeneity, domain shift, class imbalance, and distributional shifts between training and testing phases. To address this issue, we propose a hierarchical merging approach that involves local and global aggregation of models at various levels based on models' hyperparameter configurations. Furthermore, to alleviate the need for training a large number of models in the hyperparameter search, we introduce a computationally efficient method using a cyclical learning rate scheduler to produce multiple models for aggregation in the weight space. Our method demonstrates significant improvements over the model souping approach across multiple datasets (around 6% gain in HAM10000 and CheXpert datasets) while maintaining low computational costs for model generation and selection. Moreover, we achieve better results on OOD datasets than model soups. The code is available at https://github.com/BioMedIA-MBZUAI/FissionFusion.

翻译：医学数据集的标注稀缺性要求利用更广泛数据集（如ImageNet）或预训练模型（如CLIP）进行迁移学习。模型融合方法通过对多个微调模型取平均，旨在提升域内（ID）任务性能并增强对分布外（OOD）数据集的鲁棒性。然而，将这些方法应用于医学影像领域面临挑战且性能欠优，主要原因在于数据复杂性（如异质性、域偏移、类别不平衡及训练与测试阶段分布偏移）导致的误差曲面特征差异。为此，我们提出一种层次化融合方法，基于模型超参数配置在局部与全局层面进行多级聚合。此外，为缓解超参数搜索中需训练大量模型的问题，我们引入一种计算高效的方法，利用周期性学习率调度器生成多个模型以在权重空间中进行聚合。我们的方法在多个数据集上较模型融合方法有显著提升（在HAM10000与CheXpert数据集上约提升6%），同时保持模型生成与选择的低计算成本。此外，在OOD数据集上，我们获得了优于模型融合的结果。代码已开源至 https://github.com/BioMedIA-MBZUAI/FissionFusion。