Despite the success of two-stage few-shot classification methods, in the episodic meta-training stage, the model suffers severe overfitting. We hypothesize that it is caused by over-discrimination, i.e., the model learns to over-rely on the superficial features that fit for base class discrimination while suppressing the novel class generalization. To penalize over-discrimination, we introduce knowledge distillation techniques to keep novel generalization knowledge from the teacher model during training. Specifically, we select the teacher model as the one with the best validation accuracy during meta-training and restrict the symmetric Kullback-Leibler (SKL) divergence between the output distribution of the linear classifier of the teacher model and that of the student model. This simple approach outperforms the standard meta-training process. We further propose the Nearest Neighbor Symmetric Kullback-Leibler (NNSKL) divergence for meta-training to push the limits of knowledge distillation techniques. NNSKL takes few-shot tasks as input and penalizes the output of the nearest neighbor classifier, which possesses an impact on the relationships between query embedding and support centers. By combining SKL and NNSKL in meta-training, the model achieves even better performance and surpasses state-of-the-art results on several benchmarks.

翻译：尽管两阶段少样本分类方法取得了成功，但在少样本元训练阶段，模型面临着严重的过拟合问题。我们推测这是由于过度判别（over-discrimination）导致的，即模型学会了过度依赖那些有助于基类判别而抑制新类泛化的浅层特征。为了惩罚过度判别，我们引入知识蒸馏技术，在训练过程中保留来自教师模型的新类泛化知识。具体来说，我们选择在元训练过程中验证准确率最高的模型作为教师模型，并限制教师模型线性分类器的输出分布与学生模型输出分布之间的对称KL散度（SKL）。这种简单方法超越了标准的元训练过程。我们进一步提出最近邻对称KL散度（NNSKL）用于元训练，以推动知识蒸馏技术的极限。NNSKL以少样本任务为输入，惩罚最近邻分类器的输出，该输出对查询嵌入与支持中心之间的关系具有重要影响。通过在元训练中结合SKL和NNSKL，模型实现了更优的性能，并在多个基准数据集上超越了当前最先进的结果。