Retinal image matching plays a crucial role in monitoring disease progression and treatment response. However, datasets with matched keypoints between temporally separated pairs of images are not available in abundance to train transformer-based model. We propose a novel approach based on reverse knowledge distillation to train large models with limited data while preventing overfitting. Firstly, we propose architectural modifications to a CNN-based semi-supervised method called SuperRetina that help us improve its results on a publicly available dataset. Then, we train a computationally heavier model based on a vision transformer encoder using the lighter CNN-based model, which is counter-intuitive in the field knowledge-distillation research where training lighter models based on heavier ones is the norm. Surprisingly, such reverse knowledge distillation improves generalization even further. Our experiments suggest that high-dimensional fitting in representation space may prevent overfitting unlike training directly to match the final output. We also provide a public dataset with annotations for retinal image keypoint detection and matching to help the research community develop algorithms for retinal image applications.

翻译：视网膜图像匹配在监测疾病进展和治疗响应中起着关键作用。然而，缺乏足够的具有时域间隔图像对间匹配关键点标注的数据集来训练基于Transformer的模型。我们提出了一种基于反向知识蒸馏的新方法，用于在有限数据下训练大模型，同时防止过拟合。首先，我们对一种名为SuperRetina的基于CNN的半监督方法进行架构改进，从而提升其在公开数据集上的性能。随后，我们利用更轻量的CNN模型训练一个基于视觉Transformer编码器的计算密集型模型——这与知识蒸馏领域中通常使用大模型训练轻量模型的常规做法相反。令人惊讶的是，这种反向知识蒸馏进一步提升了泛化能力。实验表明，在表示空间中进行高维度的拟合可能比直接匹配最终输出更能防止过拟合。我们还提供了一个带有视网膜图像关键点检测与匹配标注的公开数据集，以帮助研究社区开发视网膜图像应用算法。