The synergy of long-range dependencies from transformers and local representations of image content from convolutional neural networks (CNNs) has led to advanced architectures and increased performance for various medical image analysis tasks due to their complementary benefits. However, compared with CNNs, transformers require considerably more training data, due to a larger number of parameters and an absence of inductive bias. The need for increasingly large datasets continues to be problematic, particularly in the context of medical imaging, where both annotation efforts and data protection result in limited data availability. In this work, inspired by the human decision-making process of correlating new evidence with previously memorized experience, we propose a Memorizing Vision Transformer (MoViT) to alleviate the need for large-scale datasets to successfully train and deploy transformer-based architectures. MoViT leverages an external memory structure to cache history attention snapshots during the training stage. To prevent overfitting, we incorporate an innovative memory update scheme, attention temporal moving average, to update the stored external memories with the historical moving average. For inference speedup, we design a prototypical attention learning method to distill the external memory into smaller representative subsets. We evaluate our method on a public histology image dataset and an in-house MRI dataset, demonstrating that MoViT applied to varied medical image analysis tasks, can outperform vanilla transformer models across varied data regimes, especially in cases where only a small amount of annotated data is available. More importantly, MoViT can reach a competitive performance of ViT with only 3.0% of the training data.

翻译：Transformer的长程依赖能力与卷积神经网络（CNN）对图像内容的局部表征因其互补优势而产生协同效应，推动了多种医学图像分析任务的架构创新与性能提升。然而，相较于CNN，Transformer因参数量更大且缺乏归纳偏置，需要更多训练数据。对日益庞大数据集的需求持续构成挑战，尤其在医学影像领域——标注成本与数据保护共同导致可用数据受限。受人类将新证据与先前记忆经验关联的决策过程启发，本文提出记忆型视觉Transformer（MoViT），旨在缓解Transformer架构成功训练与部署对大规模数据集的依赖。MoViT通过构建外部记忆结构，在训练阶段缓存历史注意力快照。为预防过拟合，我们创新性地引入记忆更新机制——注意力时域移动平均法，以历史移动平均值更新存储的外部记忆。针对推理加速，我们设计原型注意力学习方法，将外部记忆蒸馏为代表性更强的子集。在公开组织学图像数据集与内部MRI数据集上的评估表明，MoViT可适用于多种医学图像分析任务，在不同数据规模下均优于原始Transformer模型，尤其在小样本标注场景中表现突出。更重要的是，MoViT仅需3.0%的训练数据即可达到ViT的竞争性性能。