Representational transfer from publicly available models is a promising technique for improving medical image classification, especially in long-tailed datasets with rare diseases. However, existing methods often overlook the frequency-dependent behavior of these models, thereby limiting their effectiveness in transferring representations and generalizations to rare diseases. In this paper, we propose FoPro-KD, a novel framework that leverages the power of frequency patterns learned from frozen pre-trained models to enhance their transferability and compression, presenting a few unique insights: 1) We demonstrate that leveraging representations from publicly available pre-trained models can substantially improve performance, specifically for rare classes, even when utilizing representations from a smaller pre-trained model. 2) We observe that pre-trained models exhibit frequency preferences, which we explore using our proposed Fourier Prompt Generator (FPG), allowing us to manipulate specific frequencies in the input image, enhancing the discriminative representational transfer. 3) By amplifying or diminishing these frequencies in the input image, we enable Effective Knowledge Distillation (EKD). EKD facilitates the transfer of knowledge from pre-trained models to smaller models. Through extensive experiments in long-tailed gastrointestinal image recognition and skin lesion classification, where rare diseases are prevalent, our FoPro-KD framework outperforms existing methods, enabling more accessible medical models for rare disease classification. Code is available at https://github.com/xmed-lab/FoPro-KD.

翻译：从公开可用模型中迁移表征是改进医学图像分类（尤其是在存在罕见疾病的长尾数据集中）的一种有前景的技术。然而，现有方法常忽略这些模型的频率依赖行为，从而限制了其在表征迁移和泛化到罕见疾病方面的有效性。本文提出FoPro-KD——一种全新框架，利用冻结预训练模型所学的频率模式来增强其可迁移性与压缩性，并呈现若干独特见解：1）我们证明，利用公开预训练模型的表征可显著提升性能（尤其针对罕见类别），即使仅使用较小预训练模型的表征亦能奏效。2）我们发现预训练模型具有频率偏好，并通过所提出的傅里叶提示生成器（FPG）对此进行探索，从而可操控输入图像中的特定频率，增强判别性表征迁移。3）通过增强或减弱输入图像中的这些频率，我们实现了高效知识蒸馏（EKD）。EKD促进知识从预训练模型向较小模型的迁移。在罕见疾病频发的长尾胃肠道图像识别与皮肤病变分类的广泛实验中，我们的FoPro-KD框架优于现有方法，使面向罕见疾病分类的医学模型更加易于获取。代码见https://github.com/xmed-lab/FoPro-KD。