Large vision-language models have achieved outstanding performance, but their size and computational requirements make their deployment on resource-constrained devices and time-sensitive tasks impractical. Model distillation, the process of creating smaller, faster models that maintain the performance of larger models, is a promising direction towards the solution. This paper investigates the distillation of visual representations in large teacher vision-language models into lightweight student models using a small- or mid-scale dataset. Notably, this study focuses on open-vocabulary out-of-distribution (OOD) generalization, a challenging problem that has been overlooked in previous model distillation literature. We propose two principles from vision and language modality perspectives to enhance student's OOD generalization: (1) by better imitating teacher's visual representation space, and carefully promoting better coherence in vision-language alignment with the teacher; (2) by enriching the teacher's language representations with informative and finegrained semantic attributes to effectively distinguish between different labels. We propose several metrics and conduct extensive experiments to investigate their techniques. The results demonstrate significant improvements in zero-shot and few-shot student performance on open-vocabulary out-of-distribution classification, highlighting the effectiveness of our proposed approaches. Poster: https://xuanlinli17.github.io/pdfs/iccv23_large_vlm_distillation_poster.pdf Code: https://github.com/xuanlinli17/large_vlm_distillation_ood

翻译：大型视觉语言模型已取得卓越性能，但其规模和计算需求使其难以部署在资源受限设备及时间敏感型任务上。模型蒸馏——创建能维持大型模型性能的更小、更快模型的过程——是迈向解决方案的有效方向。本文研究如何利用中小规模数据集，将大型教师视觉语言模型中的视觉表征蒸馏至轻量级学生模型。值得注意的是，本研究聚焦于开放词汇分布外（OOD）泛化这一具有挑战性的问题，而该问题在既往模型蒸馏文献中常被忽视。我们从视觉与语言模态两个角度提出两项原则以增强学生的OOD泛化能力：（1）通过更好地模仿教师的视觉表征空间，并谨慎促进视觉-语言对齐中与教师的一致性；（2）通过向教师语言表征中注入信息丰富且细粒度的语义属性，以有效区分不同标签。我们提出多项评估指标并开展广泛实验以验证这些技术。结果表明，在开放词汇分布外分类任务中，学生的零样本与少样本性能取得显著提升，凸显了所提方法的有效性。海报: https://xuanlinli17.github.io/pdfs/iccv23_large_vlm_distillation_poster.pdf 代码: https://github.com/xuanlinli17/large_vlm_distillation_ood