Retinal image matching plays a crucial role in monitoring disease progression and treatment response. However, datasets with matched keypoints between temporally separated pairs of images are not available in abundance to train transformer-based model. We propose a novel approach based on reverse knowledge distillation to train large models with limited data while preventing overfitting. Firstly, we propose architectural modifications to a CNN-based semi-supervised method called SuperRetina that help us improve its results on a publicly available dataset. Then, we train a computationally heavier model based on a vision transformer encoder using the lighter CNN-based model, which is counter-intuitive in the field knowledge-distillation research where training lighter models based on heavier ones is the norm. Surprisingly, such reverse knowledge distillation improves generalization even further. Our experiments suggest that high-dimensional fitting in representation space may prevent overfitting unlike training directly to match the final output. We also provide a public dataset with annotations for retinal image keypoint detection and matching to help the research community develop algorithms for retinal image applications.

翻译：视网膜图像匹配在监测疾病进展和治疗反应中起着关键作用。然而，由于缺乏时间分离图像对之间的匹配关键点数据集，基于Transformer的模型难以充分训练。本文提出了一种基于逆向知识蒸馏的新方法，使其能在有限数据下训练大模型，同时防止过拟合。首先，我们对名为SuperRetina的CNN半监督方法进行了架构改进，从而在公开数据集上提升了其性能。随后，我们利用更轻量的CNN模型训练了一个基于视觉Transformer编码器的计算密集型模型，这与知识蒸馏研究领域中通常利用重模型训练轻量模型的做法相悖。令人惊讶的是，这种逆向知识蒸馏进一步提升了泛化能力。实验表明，与直接训练以匹配最终输出不同，在表示空间中进行高维拟合可能有助于防止过拟合。我们还提供了一个带有视网膜图像关键点检测与匹配注释的公共数据集，以帮助研究社区开发面向视网膜图像应用的算法。