The popularity of large-scale pre-training has promoted the development of medical foundation models. However, some studies have shown that although foundation models exhibit strong general feature extraction capabilities, their performance on specific tasks is still inferior to task-specific methods. In this paper, we explore a new perspective called ``Knowledge Decomposition'' to improve the performance on specific medical tasks, which deconstruct the foundation model into multiple lightweight expert models, each dedicated to a particular task, with the goal of improving specialization while concurrently mitigating resource expenditure. To accomplish the above objective, we design a novel framework named Low-Rank Knowledge Decomposition (LoRKD), which explicitly separates graidents by incorporating low-rank expert modules and the efficient knowledge separation convolution. Extensive experimental results demonstrate that the decomposed models perform well in terms of performance and transferability, even surpassing the original foundation models.

翻译：大规模预训练的普及推动了医学基础模型的发展。然而，部分研究表明，尽管基础模型展现出强大的通用特征提取能力，但其在特定任务上的性能仍逊于任务专用方法。本文探索了一种名为“知识分解”的新视角，旨在提升特定医学任务的表现——将基础模型解构为多个轻量级专家模型，各模型专注单一任务，目标是在增强专业化能力的同时降低资源消耗。为实现上述目标，我们设计了一种名为低秩知识分解（LoRKD）的新框架，该框架通过引入低秩专家模块与高效知识分离卷积，显式分离梯度。大量实验结果表明，分解后的模型在性能与可迁移性方面表现优异，甚至超越了原始基础模型。