Despite the success of two-stage few-shot classification methods, in the episodic meta-training stage, the model suffers severe overfitting. We hypothesize that it is caused by over-discrimination, i.e., the model learns to over-rely on the superficial features that fit for base class discrimination while suppressing the novel class generalization. To penalize over-discrimination, we introduce knowledge distillation techniques to keep novel generalization knowledge from the teacher model during training. Specifically, we select the teacher model as the one with the best validation accuracy during meta-training and restrict the symmetric Kullback-Leibler (SKL) divergence between the output distribution of the linear classifier of the teacher model and that of the student model. This simple approach outperforms the standard meta-training process. We further propose the Nearest Neighbor Symmetric Kullback-Leibler (NNSKL) divergence for meta-training to push the limits of knowledge distillation techniques. NNSKL takes few-shot tasks as input and penalizes the output of the nearest neighbor classifier, which possesses an impact on the relationships between query embedding and support centers. By combining SKL and NNSKL in meta-training, the model achieves even better performance and surpasses state-of-the-art results on several benchmarks.

翻译：尽管两阶段少样本分类方法取得了成功，但在情景式元训练阶段，模型会遭受严重的过拟合。我们假设这是由过度判别（即模型学习过度依赖适合基类判别的浅层特征，同时抑制新类泛化）所致。为惩罚过度判别，我们引入知识蒸馏技术，在训练过程中保留教师模型中的新类泛化知识。具体而言，我们选择元训练期间验证准确率最高的模型作为教师模型，并约束教师模型线性分类器输出分布与学生模型对应输出分布之间的对称KL散度（SKL）。这种简单方法优于标准元训练流程。我们进一步提出面向元训练的最近邻对称KL散度（NNSKL），以拓展知识蒸馏技术的潜力。NNSKL以少样本任务为输入，惩罚最近邻分类器的输出——该输出直接影响查询嵌入与支持集中心之间的关系。通过在元训练中结合SKL与NNSKL，模型性能进一步提升，并在多个基准上超越当前最优结果。