Knowledge distillation optimises a smaller student model to behave similarly to a larger teacher model, retaining some of the performance benefits. While this method can improve results on in-distribution examples, it does not necessarily generalise to out-of-distribution (OOD) settings. We investigate two complementary methods for improving the robustness of the resulting student models on OOD domains. The first approach augments the distillation with generated unlabelled examples that match the target distribution. The second method upsamples data points among the training set that are similar to the target distribution. When applied on the task of natural language inference (NLI), our experiments on MNLI show that distillation with these modifications outperforms previous robustness solutions. We also find that these methods improve performance on OOD domains even beyond the target domain.

翻译：知识蒸馏通过优化较小的学生模型使其行为与较大的教师模型相似，从而保留部分性能优势。虽然该方法能提升分布内样本的性能，但未必能泛化至分布外（OOD）场景。本研究探索两种互补方法以提升学生模型在OOD领域的鲁棒性：第一种方法通过生成与目标分布匹配的未标注样本来增强蒸馏过程；第二种方法对训练集中与目标分布相似的数据点进行上采样。在自然语言推理（NLI）任务中，基于MNLI数据集的实验表明，采用这些改进的蒸馏方法超越了现有鲁棒性解决方案。研究还发现，这些方法不仅能提升目标领域的性能，还能改善其他OOD领域的表现。